Measuring Multivariate Redundant Information with Pointwise Common Change in Surprisal
Institute of Neuroscience and Psychology, University of Glasgow, Glasgow G12 8QB, UK
Received: 2 May 2017 / Revised: 25 June 2017 / Accepted: 27 June 2017 / Published: 29 June 2017
The problem of how to properly quantify redundant information is an open question that has been the subject of much recent research. Redundant information refers to information about a target variable S
that is common to two or more predictor variables
. It can be thought of as quantifying overlapping information content or similarities in the representation of S
. We present a new measure of redundancy which measures the common change in surprisal shared between variables at the local or pointwise level. We provide a game-theoretic operational definition of unique information, and use this to derive constraints which are used to obtain a maximum entropy distribution. Redundancy is then calculated from this maximum entropy distribution by counting only those local co-information terms which admit an unambiguous interpretation as redundant information. We show how this redundancy measure can be used within the framework of the Partial Information Decomposition (PID) to give an intuitive decomposition of the multivariate mutual information into redundant, unique and synergistic contributions. We compare our new measure to existing approaches over a range of example systems, including continuous Gaussian variables. Matlab code for the measure is provided, including all considered examples.
This is an open access article distributed under the Creative Commons Attribution License
which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited (CC BY 4.0).
Printed Edition Available!
A printed edition of this Special Issue is available here
Share & Cite This Article
MDPI and ACS Style
Ince, R.A.A. Measuring Multivariate Redundant Information with Pointwise Common Change in Surprisal. Entropy 2017, 19, 318.
Ince RAA. Measuring Multivariate Redundant Information with Pointwise Common Change in Surprisal. Entropy. 2017; 19(7):318.
Ince, Robin A.A. 2017. "Measuring Multivariate Redundant Information with Pointwise Common Change in Surprisal." Entropy 19, no. 7: 318.
Show more citation formats
Show less citations formats
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.
[Return to top]
For more information on the journal statistics, click here
Multiple requests from the same IP address are counted as one view.