# Partial and Entropic Information Decompositions of a Neuronal Modulatory Interaction

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Notation and Definitions

## 3. An Interaction Designed to Be Modulatory

- If the integrated RF input is extremely weak, then the value of the transfer function is close to zero.
- If the integrated CF input is extremely weak, then the value of the transfer function should be close to the integrated RF input.
- If the integrated RF and CF inputs have the same sign, then the absolute value of the transfer function should be greater than when based on the RF input alone. On the other hand, if the RF and CF inputs are of opposite sign then the absolute value of the transfer function should be less than when based on the RF input alone.
- The sign of the value of the transfer function is that of the integrated RF, so that the context cannot change the sign of the conditional mean of the output.

- M1:
- If the $RF$ signal is strong enough and the CF input is extremely weak then $I[Y;{X}_{1}|{X}_{2}]$ can have its maximum value, $I[Y;{X}_{1}]$ can be maximised and $I[Y;{X}_{2}|{X}_{1}]$ is close to zero. This shows that the RF input is sufficient, thus allowing the information in the $RF$ to be transmitted, and that the CF input is not necessary.
- M2:
- $I[Y;{X}_{2}|{X}_{1}]$ and $I[Y;{X}_{1}]$ are close to zero when the RF input is extremely weak no matter how strong the CF input. This shows that the RF input is necessary for information to be transmitted, and that the CF input is not sufficient to transmit the information in the RF input.
- M3:
- When ${s}_{1}<{s}_{2}$ and when the RF input is weak, $I[Y;{X}_{1}]$ and $I[Y;{X}_{1}|{X}_{2}]$ are both larger when the CF input is moderate than when the CF input is weak. Thus the CF input modulates the transmission of information about the RF input.

- (i)
- the modulatory input affects output only when the primary driving integrated RF input is non-zero but weak;
- (ii)
- that even when it does have an effect it has no effect on the sign of the conditional mean output, and
- (iii)
- that it can have those modulatory effects without the binary output transmitting any unique information about the modulator.

#### 3.1. Analysis Using Classical Shannon Measures

**Theorem**

**1.**

- (a)
- $I[Y;{X}_{1}|{X}_{2}]=h\left(w\right)-2\lambda h\left(u\right)-2\mu h\left(v\right);$
- (b)
- $I[Y;{X}_{2}|{X}_{1}]=h\left(z\right)-2\lambda h\left(u\right)-2\mu h\left(v\right);$
- (c)
- $I[Y;{X}_{1}]=1-h\left(z\right);$
- (d)
- $I[Y;{X}_{2}]=1-h\left(w\right);$
- (e)
- $I[Y;{X}_{1};{X}_{2}]=1-h\left(z\right)-h\left(w\right)+2\lambda h\left(u\right)+2\mu h\left(v\right);$
- (f)
- $I[Y;({X}_{1},{X}_{2})]=1-2\lambda h\left(u\right)-2\mu h\left(v\right),$

## 4. Information Decompositions

$\mathrm{Unq}{X}_{1}\equiv {I}_{unq}[Y;{X}_{1}|{X}_{2}]$ | denotes the unique information that ${X}_{1}$ conveys about Y; |

$\mathrm{Unq}{X}_{2}\equiv {I}_{unq}[Y;{X}_{2}|{X}_{1}]$ | is the unique information that ${X}_{2}$ conveys about Y; |

${\mathrm{Shar}}_{\mathrm{S}+\mathrm{M}}\equiv {I}_{shdS+M}[Y;({X}_{1},{X}_{2})]$ | gives the common (or redundant or shared) information that both ${X}_{1}$ and ${X}_{2}$ have about Y; |

$\mathrm{Syn}\equiv {I}_{syn}[Y;({X}_{1},{X}_{2})]$ | is the synergy or information that the joint variable $({X}_{1},{X}_{2})$ has about Y that cannot be obtained by observing ${X}_{1}$ and ${X}_{2}$ separately. |

#### 4.1. The Ibroja PID

#### 4.2. The EID Using ${I}_{\mathrm{ccs}}$

## 5. Information Decomposition (ID) Spectra

#### 5.1. Definition and Illustrations

#### 5.2. Ibroja Spectra

#### 5.3. EID Spectra

#### 5.4. Contextual Modulation and Information Decompositions

- S1:
- If the $RF$ signal is strong enough, and the CF input is extremely weak, then both UnqX2 and Syn are close to zero, UnqX1 can have its maximum value, and the sum of UnqX1 and ${\mathrm{Shar}}_{\mathrm{S}+\mathrm{M}}$ can equal the total output entropy. This shows that the RF input is sufficient, thus allowing the information in the $RF$ to be transmitted, and that the CF input is not necessary.
- S2:
- All five partial information components are close to zero when the RF input is extremely weak no matter how strong the CF input. This shows that the RF input is necessary for information to be transmitted, and that the CF input is not sufficient to transmit the information in the RF input.
- S3:
- When ${s}_{1}<{s}_{2}$ and when the RF input is weak, then the sum of UnqX1 and Syn is larger when the CF input is moderate than it is when the CF input is weak. The same is true of the sum of UnqX1 and ${\mathrm{Shar}}_{\mathrm{S}+\mathrm{M}}$. Thus the CF input modulates the transmission of information about the RF input.

- S1’:
- When UnqX2 < 0, UnqX2 and Syn are approximately of the same magnitude, the sum of UnqX1 and Syn can have its maximum value, and the sum of UnqX1 and ${\mathrm{Shar}}_{\mathrm{S}+\mathrm{M}}$ can equal the total output entropy.
- S2’:
- If at least one component is negative, then we can set the left-hand sides of (18)–(21) to zero and use the rule that the sum of the magnitudes of the negative components is approximately equal to the sum of the magnitudes of the positive components. If in any of (18)–(21) there is no negative term then all terms on the right-hand side are close to zero.

#### 5.5. Comparison of PID and EID

## 6. Analysis of the Transfer Functions Using the Ibroja PID over a Wide Range of Input Strengths

**Theorem**

**2.**

**Corollary**

**1.**

**Corollary**

**2.**

**Theorem**

**3.**

**Theorem**

**4.**

- (a)
- When transfer function ${T}_{M}$ is employed then
- (i)
- Syn $=I(Y;{X}_{2}|{X}_{1})=h\left({z}_{M}\right)-2\lambda h\left({u}_{M}\right)-2\mu h\left({v}_{M}\right)$;
- (ii)
- ${Shar}_{S+M}=I(Y;{X}_{2})=1-h\left({w}_{M}\right)$;
- (iii)
- Unq${X}_{1}=I(Y;{X}_{1}|{X}_{2})-I(Y;{X}_{2}|{X}_{1})=h\left({w}_{M}\right)-h\left({z}_{M}\right),\phantom{\rule{1.em}{0ex}}and\phantom{\rule{1.em}{0ex}}$ Unq${X}_{2}=0$.

- (b)
- When the transfer function ${T}_{A}$ is used and ${s}_{1}={s}_{2}$ then
- (i)
- Syn $=I(Y;{X}_{2}|{X}_{1})=h\left({z}_{A}\right)-2\lambda h\left({u}_{A}\right)-2\mu ;$
- (ii)
- ${Shar}_{S+M}=I(Y;{X}_{1})=1-h\left({z}_{A}\right);$
- (iii)
- Unq${X}_{1}$ = Unq${X}_{2}=0;$

- (c)
- When the transfer function ${T}_{A}$ is used and ${s}_{1}<{s}_{2}$ then
- (i)
- Syn $=I(Y;{X}_{1}|{X}_{2})=h\left({w}_{A}\right)-2\lambda h\left({u}_{A}\right)-2\mu h\left({v}_{A}\right);$
- (ii)
- ${Shar}_{S+M}=I(Y;{X}_{1})=1-h\left({z}_{A}\right);$
- (iii)
- Unq${X}_{2}=I(Y;{X}_{2}|{X}_{1})-I(Y;{X}_{1}|{X}_{2})=h\left({z}_{A}\right)-h\left({w}_{A}\right)\phantom{\rule{1.em}{0ex}}and\phantom{\rule{1.em}{0ex}}$ Unq${X}_{1}=0.$

- (d)
- When the transfer function ${T}_{A}$ is used and ${s}_{1}>{s}_{2}$ then
- (i)
- Syn $=I(Y;{X}_{2}|{X}_{1})=h\left({z}_{A}\right)-2\lambda h\left({u}_{A}\right)-2\mu h\left({v}_{A}\right)$;
- (ii)
- ${Shar}_{S+M}=I(Y;{X}_{2})=1-h\left({w}_{A}\right)$;
- (iii)
- Unq${X}_{1}=I(Y;{X}_{1}|{X}_{2})-I(Y;{X}_{2}|{X}_{1})=h\left({w}_{A}\right)-h\left({z}_{A}\right),\phantom{\rule{1.em}{0ex}}and\phantom{\rule{1.em}{0ex}}$ Unq${X}_{2}=0$.

**Theorem**

**5.**

## 7. Analysis of the Transfer Functions Using EID over a Wide Range of Input Strengths

## 8. Applications of ID Measures to Psychophysical Data

## 9. Conclusions and Discussion

#### 9.1. Implications of These Findings for Conceptions of ‘Modulation’ in the Cognitive and Neurosciences

#### 9.2. Comparisons between PID and EID

#### 9.3. Using EID and PID to Analyze and Interpret Psychophysical Data

#### 9.4. Using ID Spectra to Analyze and Interpret Empirical Data in General

#### 9.5. Modulatory Regulation of Activity as a Crucial and Non-Trivial Aspect of Information Processing

## Acknowledgments

## Author Contributions

## Conflicts of Interest

## Appendix A. Preliminary Results

## Appendix B. Proof of Theorem 1

## Appendix C. Proof of Theorem 2

## Appendix D. Proof of Corollary 1

## Appendix E. Proof of Corollary 2

## Appendix F. Proof of Theorem 3

## Appendix G. Proof of Theorem 4

## Appendix H. Proof of Theorem 5

## References

- Gilbert, C.D.; Sigman, M. Brain States: Top-Down Influences in Sensory Processing. Neuron
**2007**, 54, 677–696. [Google Scholar] [CrossRef] [PubMed] - Phillips, W.A.; Singer, W. In search of common foundations for cortical computation. Behav. Brain Sci.
**1997**, 20, 657–722. [Google Scholar] [CrossRef] [PubMed] - Phillips, W.A.; Silverstein, S.M. Convergence of biological and psychological perspectives on cognitive coordination in schizophrenia. Behav. Brain Sci.
**2003**, 26, 65–138. [Google Scholar] [CrossRef] [PubMed] - Lamme, V.A.F. Beyond the classical receptive field: Contextual modulation of V1 responses. In The Visual Neurosciences; Werner, J.S., Chalupa, L.M., Eds.; MIT Press: Cambridge, MA, USA, 2004; pp. 720–732. [Google Scholar]
- Kay, J.; Floreano, D.; Phillips, W.A. Contextually guided unsupervised learning using local multivariate binary processors. Neural Netw.
**1998**, 11, 117–140. [Google Scholar] [CrossRef] - Larkum, M. A cellular mechanism for cortical associations: An organizing principle for the cerebral cortex. Trends Neurosci.
**2013**, 36, 141–151. [Google Scholar] [CrossRef] [PubMed] - Phillips, W.A.; Larkum, M.E.; Harley, C.W.; Silverstein, S.M. The effects of arousal on apical amplification and conscious state. Neurosci. Conscious.
**2016**, 1–13. [Google Scholar] [CrossRef] - Williams, P.L.; Beer, R.D. Nonnegative Decomposition of Multivariate Information. arXiv, 2010; arXiv:1004.2515. [Google Scholar]
- Bertschinger, N.; Rauh, J.; Olbrich, E.; Jost, J.; Ay, N. Quantifying Unique Information. Entropy
**2014**, 16, 2161–2183. [Google Scholar] [CrossRef] - Griffith, V.; Koch, C.; Griffith, V. Quantifying synergistic mutual information. In Guided Self-Organization: Inception. Emergence, Complexity and Computation; Springer: Berlin/Heidelberg, Germany, 2014; Volume 9, pp. 159–190. [Google Scholar]
- James, R.G.; Emenheiser, J.; Crutchfield, J.P. Unique Information via Dependency Constraints. arXiv, 2017; arXiv:1709.06653. [Google Scholar]
- Ince, R.A.A. Measuring multivariate redundant information with pointwise common change in surprisal. Entropy
**2017**, 19, 318. [Google Scholar] [CrossRef] - Ince, R.A.A. The Partial Entropy Decomposition: Decomposing multivariate entropy and mutual information via pointwise common surprisal. arXiv, 2017; arXiv:1702.01591. [Google Scholar]
- Phillips, W.A.; Kay, J.; Smyth, D. The discovery of structure by multi-stream networks of local processors with contextual guidance. Netw. Comput. Neural Syst.
**1995**, 6, 225–246. [Google Scholar] [CrossRef] - Cover, T.M.; Thomas, J.A. Elements of Information Theory; Wiley-Interscience: New York, NY, USA, 1991. [Google Scholar]
- Schneidman, E.; Bialek, W.; Berry, M.J. Synergy, Redundancy, and Population Codes. J. Neurosci.
**2003**, 23, 11539–11553. [Google Scholar] [PubMed] - Kay, J. Neural networks for unsupervised learning based on information theory. In Statistics and Neural Networks: Advances at the Interface; Kay, J.W., Titterington, D.M., Eds.; Oxford University Press: Oxford, UK, 1999; pp. 25–63. [Google Scholar]
- Kay, J.; Phillips, W.A. Activation functions, computational goals and learning rules for local processors with contextual guidance. Neural Comput.
**1997**, 9, 895–910. [Google Scholar] [CrossRef] - Kay, J.W.; Phillips, W.A. Coherent infomax as a computational goal for neural systems. Bull. Math. Biol.
**2011**, 73, 344–372. [Google Scholar] [CrossRef] [PubMed] - James, R.G.; Crutchfield, J.P. Multivariate Dependence beyond Shannon Information. Entropy
**2017**, 19, 530. [Google Scholar] [CrossRef] - Wibral, M.; Priesemann, V.; Kay, J.W.; Lizier, J.T.; Phillips, W.A. Partial information decomposition as a unified approach to the specification of neural goal functions. Brain Cognit.
**2017**, 112, 25–38. [Google Scholar] [CrossRef] [PubMed] - Harder, M.; Salge, C.; Polani, D. Bivariate measure of redundant information. Phys. Rev. E
**2013**, 87. [Google Scholar] [CrossRef] [PubMed] [Green Version] - Pica, G.; Piasini, E.; Chicharro, D.; Panzeri, S. Invariant components of synergy, redundancy, and unique information. Entropy
**2017**, 19, 451. [Google Scholar] [CrossRef] - Wibral, M.; Lizier, J.T.; Vögler, S.; Priesemann, V.; Galuske, R. Local active information storage as a tool to understand distributed neural information processing. Front. Neuroinf.
**2014**, 8. [Google Scholar] [CrossRef] [PubMed] - Lizier, J.T.; Prokopenko, M.; Zomaya, A. Local information transfer as a spatiotemporal filter for complex systems. Phys. Rev. E
**2008**, 77. [Google Scholar] [CrossRef] [PubMed] - Wibral, M.; Lizier, J.T.; Priesemann, V. Bits from brains for biologically inspired computing. Front. Robot. AI
**2015**. [Google Scholar] [CrossRef] - Van de Cruys, T. Two Multivariate Generalizations of Pointwise Mutual Information. In Proceedings of the Workshop on Distributional Semantics and Compositionality, Portland, Oregon, 24 June 2011; pp. 16–20. [Google Scholar]
- Church, K.W.; Hanks, P. Word Association Norms, Mutual Information, and Lexicography. Comput. Linguist.
**1990**, 16, 22–29. [Google Scholar] - James, R.G.; Ellison, C.J.; Crutchfield, J.P. Anatomy of a bit: Information in a time series observation. Chaos
**2011**, 037109. [Google Scholar] [CrossRef] [PubMed] - Olbrich, E.; Bertschinger, N.; Rauh, J. Information decomposition and synergy. Entropy
**2015**, 17, 3501–3517. [Google Scholar] [CrossRef] - Barrett, A.B. An exploration of synergistic and redundant information sharing in static and dynamical Gaussian systems. Phys. Rev. E
**2015**, 91, doi. [Google Scholar] [CrossRef] [PubMed] - Chen, C.C.; Kasamatsu, T.; Polat, U.; Norcia, A.M. Contrast response characteristics of long-range lateral interactions in cat striate cortex. Neuroreport
**2001**, 12, 655–661. [Google Scholar] [CrossRef] [PubMed] - Polat, U.; Mizobe, K.; Pettet, M.W.; Kasamatsu, T.; Norcia, A.M. Collinear stimuli regulate visual responses depending on cell’s contrast threshold. Nature
**1998**, 391, 580–584. [Google Scholar] [PubMed] - Ince, R.A.A.; Giordano, B.L.; Kayser, C.; Rousselet, G.A.; Gross, J.; Schyns, P.G. A Statistical Framework for Neuroimaging Data Analysis Based on Mutual Information Estimated via a Gaussian Copula. Hum. Brain Mapp.
**2017**, 38, 1541–1573. [Google Scholar] [CrossRef] [PubMed] - Panzeri, S.; Senatore, R.; Montemurro, M.A.; Petersen, R.S. Correcting for the Sampling Bias Problem in Spike Train Information Measures. J. Neurophys.
**2007**, 98, 1064–1072. [Google Scholar] [CrossRef] [PubMed] - Ince, R.A.A.; Mazzoni, A.; Bartels, A.; Logothetis, N.K.; Panzeri, S. A Novel Test to Determine the Significance of Neural Selectivity to Single and Multiple Potentially Correlated Stimulus Features. J. Neurosci. Methods
**2012**, 210, 49–65. [Google Scholar] [CrossRef] [PubMed] - Stramaglia, S.; Angelini, L.; Wu, G.; Cortes, J.; Faes, L.; Marinazzo, D. Synergistic and redundant information flow detected by unnormalized Granger causality: Application to resting state fMRI. IEEE Trans. Biomed. Eng.
**2016**, 63, 2518–2524. [Google Scholar] [CrossRef] [PubMed] - Timme, N.M.; Ito, S.; Myroshnychenko, M.; Nigam, S.; Shimono, M.; Yeh, F.-C. High-Degree Neurons Feed Cortical Computations. PLoS Comput. Biol.
**2016**, 12, e1004858. [Google Scholar] [CrossRef] [PubMed] - Phillips, W.A.; Clark, A.; Silverstein, S.M. On the functions, mechanisms, and malfunctions of intracortical contextual modulation. Neurosci. Biobehav. Rev.
**2015**, 52, 1–20. [Google Scholar] [CrossRef] [PubMed]

**Figure 1.**A local processor with binary receptive field (RF) input ${X}_{1},$ contextual field (CF) input ${X}_{2}$ and output Y. The weights on the connections from the RF and CF inputs into the output unit are ${s}_{1},{s}_{2},$ which represent the strengths given to the input signals. The integrated RF input, r, and the integrated CF input, c, are passed through a transfer function T and a logistic nonlinearity within the output unit to produce the conditional output probability, $\theta $, as well as the output conditional mean, m.

**Figure 2.**Classical Shannon measures (in bits), based on additive and modulatory transfer functions, and a correlation between inputs of 0.78.

**Figure 3.**Classical Shannon measures (in bits), based on additive and modulatory transfer functions, and a zero correlation between inputs.

**Figure 4.**Partial information decomposition (PID) and entropic information decomposition (EID) spectra (in bits), based on additive (A) and modulatory (M) transfer functions for four combinations of signal strengths: 1. (${s}_{1}=10.0,{s}_{2}=0.05$), 2. (${s}_{1}=0.05,{s}_{2}=10.0$), 3. (${s}_{1}=1.0,{s}_{2}=0.05$), 4. (${s}_{1}=1.0,{s}_{2}=5.0$), and two values of the correlation between inputs: 0.78 and zero.

**Figure 5.**Ibroja surfaces, based on additive and modulatory transfer functions, and a correlation between inputs of 0.78.

**Figure 6.**Ibroja surfaces, based on additive and modulatory transfer functions, and zero correlation between inputs.

**Figure 7.**EID surfaces, based on additive and modulatory transfer functions, and a correlation between inputs of $0.78$.

**Figure 8.**EID surfaces, based on additive and modulatory transfer functions, and a zero correlation between inputs.

**Figure 9.**Examples of gabor patch stimuli used in the psychophysical experiment. In all conditions, the task was to detect the presence of a centrally presented target gabor.

**Figure 10.**Partial information decomposition (PID) and EID spectra (in bits) calculated for subject 10 (S10), subject 18 (S18) and the whole group of subjects (G) in the contrast detection experiment calculated at threshold (AT) and over threshold (OT).

**Table 1.**Estimated accuracy, with estimated standard error, for each combination of the three conditions and the absence or presence of flankers.

No Target | At Threshold | Over Threshold | |
---|---|---|---|

Without Flankers | 0.9096 (0.0273) | 0.8797 (0.0289) | 0.9824 (0.0037) |

With Flankers | 0.9629 (0.0150) | 0.3766 (0.0532) | 0.9849 (0.0039) |

© 2017 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Kay, J.W.; Ince, R.A.A.; Dering, B.; Phillips, W.A.
Partial and Entropic Information Decompositions of a Neuronal Modulatory Interaction. *Entropy* **2017**, *19*, 560.
https://doi.org/10.3390/e19110560

**AMA Style**

Kay JW, Ince RAA, Dering B, Phillips WA.
Partial and Entropic Information Decompositions of a Neuronal Modulatory Interaction. *Entropy*. 2017; 19(11):560.
https://doi.org/10.3390/e19110560

**Chicago/Turabian Style**

Kay, Jim W., Robin A. A. Ince, Benjamin Dering, and William A. Phillips.
2017. "Partial and Entropic Information Decompositions of a Neuronal Modulatory Interaction" *Entropy* 19, no. 11: 560.
https://doi.org/10.3390/e19110560