Next Article in Journal
Using k-Mix-Neighborhood Subdigraphs to Compute Canonical Labelings of Digraphs
Previous Article in Journal
The More You Know, the More You Can Grow: An Information Theoretic Approach to Growth in the Information Age
Open AccessArticle

Quantifying Synergistic Information Using Intermediate Stochastic Variables

Computational Science Lab, University of Amsterdam, 1098 XH Amsterdam, The Netherlands
The Institute for Advanced Study, University of Amsterdam, Oude Turfmarkt 147, 1012 GC Amsterdam, The Netherlands
Advanced Computing Lab, ITMO University, Kronverkskiy pr. 49, 197101 Saint Petersburg, Russia
Complexity Institute, Nanyang Technological University, 639673 Singapore, Singapore
Author to whom correspondence should be addressed.
This paper is an extended version of our paper published in the Conference on Complex Systems, Amsterdam, The Netherlands, 19–22 September 2016.
Academic Editor: J. A. Tenreiro Machado
Entropy 2017, 19(2), 85;
Received: 1 November 2016 / Revised: 16 February 2017 / Accepted: 19 February 2017 / Published: 22 February 2017
(This article belongs to the Section Complexity)
Quantifying synergy among stochastic variables is an important open problem in information theory. Information synergy occurs when multiple sources together predict an outcome variable better than the sum of single-source predictions. It is an essential phenomenon in biology such as in neuronal networks and cellular regulatory processes, where different information flows integrate to produce a single response, but also in social cooperation processes as well as in statistical inference tasks in machine learning. Here we propose a metric of synergistic entropy and synergistic information from first principles. The proposed measure relies on so-called synergistic random variables (SRVs) which are constructed to have zero mutual information about individual source variables but non-zero mutual information about the complete set of source variables. We prove several basic and desired properties of our measure, including bounds and additivity properties. In addition, we prove several important consequences of our measure, including the fact that different types of synergistic information may co-exist between the same sets of variables. A numerical implementation is provided, which we use to demonstrate that synergy is associated with resilience to noise. Our measure may be a marked step forward in the study of multivariate information theory and its numerous applications. View Full-Text
Keywords: synergy; synergistic information; synergistic entropy; information theory; stochastic variables; higher order information synergy; synergistic information; synergistic entropy; information theory; stochastic variables; higher order information
Show Figures

Figure 1

MDPI and ACS Style

Quax, R.; Har-Shemesh, O.; Sloot, P.M.A. Quantifying Synergistic Information Using Intermediate Stochastic Variables. Entropy 2017, 19, 85.

Show more citation formats Show less citations formats
Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Access Map by Country/Region

Search more from Scilit
Back to TopTop