# Efficient Algorithms for Searching the Minimum Information Partition in Integrated Information Theory

^{1}

^{2}

^{3}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Measures of Integrated Information

#### 2.1. Multi (Mutual) Information ${\mathrm{\Phi}}_{\mathrm{MI}}$

#### 2.2. Stochastic Interaction ${\mathrm{\Phi}}_{\mathrm{SI}}$

#### 2.3. Geometric Integrated Information ${\mathrm{\Phi}}_{\mathrm{G}}$

## 3. Minimum Information Partition

## 4. Submodular Optimization

#### 4.1. Submodularity

**Definition**

**1**

**.**Let Ω be a finite set and ${2}^{\mathrm{\Omega}}$ its power set. A set function $f:{2}^{\mathrm{\Omega}}\to \mathbb{R}$ is submodular if it satisfies the following inequality for any $S,T\subseteq \mathrm{\Omega}$:

#### 4.2. Queyranne’s Algorithm

#### 4.3. Submodularity in Measures of Integrated Information

## 5. Replica Exchange Markov Chain Monte Carlo Method

## 6. Results

#### 6.1. Speed of Queyranne’s Algorithm Compared With Exhaustive Search

#### 6.2. Accuracy of Queyranne’s Algorithm

**Correct rate (CR):**Correct rate (CR) is the rate of correctly finding the MIP.**Rank (RA):**Rank (RA) is the rank of the partition found by Queyranne’s algorithm among all possible partitions. The rank is based on the $\mathrm{\Phi}$ values computed at each partition. The partition that gives the lowest $\mathrm{\Phi}$ is rank 1. The highest rank is equal to the number of possible bi-partitions, ${2}^{N-1}$.**Error ratio (ER):**Error ratio (ER) is the deviation of the value of integrated information computed across the partition found by Queyranne’s algorithm from that computed across the MIP, which is normalized by the mean error computed at all possible partitions. Error ratio is defined by$$\mathrm{Error}\mathrm{Ratio}=\frac{{\mathrm{\Phi}}_{\mathrm{Q}}-{\mathrm{\Phi}}_{\mathrm{MIP}}}{\overline{\mathrm{\Phi}}-{\mathrm{\Phi}}_{\mathrm{MIP}}},$$**Correlation (CORR):**Correlation (CORR) is the correlation between the partition found by Queyranne’s algorithm and the MIP found by the exhaustive search. Let us represent a bi-partition of N-elements as an N-dimensional vector $\mathit{\sigma}=({\sigma}_{1},\dots ,{\sigma}_{N})\in {\{-1,1\}}^{N}$, where $\pm 1$ indicates one of the two subgroups. The absolute value of the correlation between the vector given by the MIP (${\mathit{\sigma}}^{\mathrm{MIP}}$) and that given by the partition found by Queyranne’s algorithm (${\mathit{\sigma}}^{\mathrm{Q}}$) is computed:$$|\mathrm{corr}({\mathit{\sigma}}^{\mathrm{MIP}},{\mathit{\sigma}}^{\mathrm{Q}})|=\left|\frac{{\sum}_{i=1}^{N}({\sigma}_{i}^{\mathrm{MIP}}-{\overline{\sigma}}^{\mathrm{MIP}})({\sigma}_{i}^{\mathrm{Q}}-{\overline{\sigma}}^{\mathrm{Q}})}{\sqrt{{\sum}_{i=1}^{N}{({\sigma}_{i}^{\mathrm{MIP}}-{\overline{\sigma}}^{\mathrm{MIP}})}^{2}{\sum}_{i=1}^{N}{({\sigma}_{i}^{\mathrm{Q}}-{\overline{\sigma}}^{\mathrm{Q}})}^{2}}}\right|,$$

#### 6.3. Comparison between Queyranne’s Algorithm and REMCMC

#### 6.4. Evaluation with Real Neural Data

## 7. Discussion

## Acknowledgments

## Author Contributions

## Conflicts of Interest

## Abbreviations

IIT | integrated information theory |

MIP | minimum information partition |

MCMC | Markov chain Monte Carlo |

REMCMC | replica exchange Markov chain Monte Carlo |

EEG | electroencephalography |

ECoG | electrocorticography |

AR | autoregressive |

CR | correct rate |

RA | rank |

ER | error ratio |

CORR | correlation |

MCS | Monte Carlo step |

## Appendix A. Analytical Formula of Φ for Gaussian Variables

#### Appendix A.1. Multi Information

#### Appendix A.2. Stochastic Interaction

#### Appendix A.3. Geometric Integrated Information

## Appendix B. Details of Replica Exchange Markov Chain Monte Carlo Method

#### Appendix B.1. Metropolis Method

**Propose a candidate of the next sample**An element e is randomly selected and if it is in the current subset ${S}^{(t)}$, the candidate ${S}_{\mathrm{c}}$ is ${S}^{(t)}\backslash \left\{e\right\}$. If not, the candidate is ${S}^{(t)}\cup \left\{e\right\}$.**Determine whether to accept the candidate or not**The candidate ${S}_{\mathrm{c}}$ is accepted (${S}^{(t+1)}={S}_{\mathrm{c}}$) or not accepted (${S}^{(t+1)}={S}^{(t)}$) according to the following probability $a({S}^{(t)}\to {S}_{\mathrm{c}})$:$$\begin{array}{cc}\hfill a({S}^{(t)}\to {S}_{\mathrm{c}})& =min(1,r),\hfill \\ \hfill r& =\frac{p({S}_{\mathrm{c}};\beta )}{p({S}^{(t)};\beta )}=exp\left[\beta \left\{\mathrm{\Phi}({S}^{(t)})-\mathrm{\Phi}({S}_{\mathrm{c}})\right\}\right].\hfill \end{array}$$This probability means that if the integrated information decreases by stepping from ${S}^{(t)}$ to ${S}_{\mathrm{c}}$, the candidate ${S}_{\mathrm{c}}$ is always accepted, and otherwise it is accepted with the probability r.

#### Appendix B.2. Replica Exchange Markov Chain Monte Carlo

**Sampling from each distribution:**Samples are drawn from each distribution $p({S}_{m};{\beta}_{m})$ separately by using the Metropolis method as described in the previous subsection.**Exchange between neighboring inverse temperatures:**After a given number of samples are drawn, subsets at neighboring inverse temperatures are swapped, according to the following probability $p({S}_{m}\leftrightarrow {S}_{m+1})$:$$\begin{array}{cc}\hfill p({S}_{m}\leftrightarrow {S}_{m+1})& =min(1,{r}^{\prime}),\hfill \\ \hfill {r}^{\prime}& =\frac{p({S}_{m+1};{\beta}_{m})p({S}_{m};{\beta}_{m+1})}{p({S}_{m};{\beta}_{m})p({S}_{m+1};{\beta}_{m+1})}\hfill \\ \hfill & =exp\left[({\beta}_{m+1}-{\beta}_{m})\left\{\mathrm{\Phi}({S}_{m+1})-\mathrm{\Phi}({S}_{m})\right\}\right].\hfill \end{array}$$This probability indicates that if the integrated information at a higher inverse temperature is larger than that at a lower inverse temperature, subsets are always swapped; otherwise, they are swapped with the probability ${r}^{\prime}$.

#### Appendix B.2.1. Initial Setting

#### Appendix B.2.2. Updating

#### Appendix B.3. Convergence Criterion

#### Appendix B.4. Parameter Settings

## Appendix C. Values of Φ

**Figure A1.**The values of $\mathrm{\Phi}$ for the block-structured models at $\sigma =0.01$. The box plots represent the distribution of $\mathrm{\Phi}$ at all the partitions. The red solid line indicates $\mathrm{\Phi}$ at the MIP. The green circles indicate $\mathrm{\Phi}$ at the partitions found by Queyranne’s algorithm. (

**a**) ${\mathrm{\Phi}}_{\mathrm{SI}}$, (

**b**) ${\mathrm{\Phi}}_{\mathrm{G}}$.

## References

- Tononi, G.; Sporns, O.; Edelman, G.M. A measure for brain complexity: Relating functional segregation and integration in the nervous system. Proc. Natl. Acad. Sci. USA
**1994**, 91, 5033–5037. [Google Scholar] [CrossRef] [PubMed] - Tononi, G. An information integration theory of consciousness. BMC Neurosci.
**2004**, 5, 42. [Google Scholar] [CrossRef] [PubMed][Green Version] - Tononi, G. Consciousness as integrated information: A provisional manifesto. Biol. Bull.
**2008**, 215, 216–242. [Google Scholar] [CrossRef] [PubMed] - Oizumi, M.; Albantakis, L.; Tononi, G. From the phenomenology to the mechanisms of consciousness: Integrated information theory 3.0. PLoS Comput. Biol.
**2014**, 10, e1003588. [Google Scholar] [CrossRef] [PubMed] - Massimini, M.; Ferrarelli, F.; Huber, R.; Esser, S.K.; Singh, H.; Tononi, G. Breakdown of cortical effective connectivity during sleep. Science
**2005**, 309, 2228–2232. [Google Scholar] [CrossRef] [PubMed] - Casali, A.G.; Gosseries, O.; Rosanova, M.; Boly, M.; Sarasso, S.; Casali, K.R.; Casarotto, S.; Bruno, M.A.; Laureys, S.; Tononi, G.; et al. A theoretically based index of consciousness independent of sensory processing and behavior. Sci. Transl. Med.
**2013**, 5, 198ra105. [Google Scholar] [CrossRef] [PubMed] - Lee, U.; Mashour, G.A.; Kim, S.; Noh, G.J.; Choi, B.M. Propofol induction reduces the capacity for neural information integration: Implications for the mechanism of consciousness and general anesthesia. Conscious. Cogn.
**2009**, 18, 56–64. [Google Scholar] [CrossRef] [PubMed] - Chang, J.Y.; Pigorini, A.; Massimini, M.; Tononi, G.; Nobili, L.; Van Veen, B.D. Multivariate autoregressive models with exogenous inputs for intracerebral responses to direct electrical stimulation of the human brain. Front. Hum. Neurosci.
**2012**, 6, 317. [Google Scholar] [CrossRef] [PubMed] - Boly, M.; Sasai, S.; Gosseries, O.; Oizumi, M.; Casali, A.; Massimini, M.; Tononi, G. Stimulus set meaningfulness and neurophysiological differentiation: A functional magnetic resonance imaging study. PLoS ONE
**2015**, 10, e0125337. [Google Scholar] [CrossRef] [PubMed] - Haun, A.M.; Oizumi, M.; Kovach, C.K.; Kawasaki, H.; Oya, H.; Howard, M.A.; Adolphs, R.; Tsuchiya, N. Conscious Perception as Integrated Information Patterns in Human Electrocorticography. eNeuro
**2017**, 4, 1–18. [Google Scholar] [CrossRef] [PubMed] - Balduzzi, D.; Tononi, G. Integrated information in discrete dynamical systems: Motivation and theoretical framework. PLoS Comput. Biol.
**2008**, 4, e1000091. [Google Scholar] [CrossRef] [PubMed] - Oizumi, M.; Tsuchiya, N.; Amari, S.i. Unified framework for information integration based on information geometry. Proc. Natl. Acad. Sci. USA
**2016**, 113, 14817–14822. [Google Scholar] [CrossRef] [PubMed] - Hidaka, S.; Oizumi, M. Fast and exact search for the partition with minimal information loss. arXiv, 2017; arXiv:1708.01444. [Google Scholar]
- Queyranne, M. Minimizing symmetric submodular functions. Math. Program.
**1998**, 82, 3–12. [Google Scholar] [CrossRef] - Barrett, A.B.; Barnett, L.; Seth, A.K. Multivariate Granger causality and generalized variance. Phys. Rev. E
**2010**, 81, 041907. [Google Scholar] [CrossRef] [PubMed] - Oizumi, M.; Amari, S.; Yanagawa, T.; Fujii, N.; Tsuchiya, N. Measuring integrated information from the decoding perspective. PLoS Comput. Biol.
**2016**, 12, e1004654. [Google Scholar] [CrossRef] [PubMed] - Tegmark, M. Improved measures of integrated information. PLoS Comput. Biol.
**2016**, 12, e1005123. [Google Scholar] [CrossRef] [PubMed] - Ay, N. Information geometry on complexity and stochastic interaction. MIP MIS Preprint 95
**2001**. Available online: http://www.mis.mpg.de/publications/preprints/2001/prepr2001-95.html (accessed on 6 March 2018). - Ay, N. Information geometry on complexity and stochastic interaction. Entropy
**2015**, 17, 2432–2458. [Google Scholar] [CrossRef] - Amari, S.; Tsuchiya, N.; Oizumi, M. Geometry of information integration. arXiv, 2017; arXiv:1709.02050. [Google Scholar]
- Swendsen, R.H.; Wang, J.S. Replica Monte Carlo simulation of spin-glasses. Phys. Rev. Lett.
**1986**, 57, 2607–2609. [Google Scholar] [CrossRef] [PubMed] - Geyer, C.J. Markov chain Monte Carlo maximum likelihood. In Proceedings of the 23rd Symposium on the Interface, Seattle, WA, USA, 21–24 April 1991; Interface Foundation of North America: Fairfax Station, VA, USA, 1991; pp. 156–163. [Google Scholar]
- Hukushima, K.; Nemoto, K. Exchange Monte Carlo method and application to spin glass simulations. J. Phys. Soc. Jpn.
**1996**, 65, 1604–1608. [Google Scholar] [CrossRef] - Earl, D.J.; Deem, M.W. Parallel tempering: Theory, applications, and new perspectives. Phys. Chem. Chem. Phys.
**2005**, 7, 3910–3916. [Google Scholar] [CrossRef] [PubMed] - Burnham, K.P.; Anderson, D.R. Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach; Springer: New York, NY, USA, 2003. [Google Scholar]
- Watanabe, S. Information theoretical analysis of multivariate correlation. IBM J. Res. Dev.
**1960**, 4, 66–82. [Google Scholar] [CrossRef] - Studený, M.; Vejnarová, J. The Multiinformation Function as a Tool For Measuring Stochastic Dependence; MIT Press: Cambridge, MA, USA, 1999. [Google Scholar]
- Pearl, J. Causality; Cambridge University Press: Cambridge, UK, 2009. [Google Scholar]
- Iwata, S. Submodular function minimization. Math. Program.
**2008**, 112, 45–64. [Google Scholar] [CrossRef] - Wishart, J. The generalised product moment distribution in samples from a normal multivariate population. Biometrika
**1928**, 20A, 32–52. [Google Scholar] [CrossRef] - Bishop, C.M. Pattern Recognition and Machine Learning; Springer: New York, NY, USA, 2006. [Google Scholar]
- Pinn, K.; Wieczerkowski, C. Number of magic squares from parallel tempering Monte Carlo. Int. J. Mod. Phys. C
**1998**, 9, 541–546. [Google Scholar] [CrossRef] - Hukushima, K. Extended ensemble Monte Carlo approach to hardly relaxing problems. Computer Phys. Commun.
**2002**, 147, 77–82. [Google Scholar] [CrossRef] - Nagata, K.; Kitazono, J.; Nakajima, S.; Eifuku, S.; Tamura, R.; Okada, M. An Exhaustive Search and Stability of Sparse Estimation for Feature Selection Problem. IPSJ Online Trans.
**2015**, 8, 25–32. [Google Scholar] [CrossRef] - Nagasaka, Y.; Shimoda, K.; Fujii, N. Multidimensional recording (MDR) and data sharing: an ecological open research and educational platform for neuroscience. PLoS ONE
**2011**, 6, e22561. [Google Scholar] [CrossRef] [PubMed] - Toker, D.; Sommer, F. Information Integration in Large Brain Networks. arXiv, 2017; arXiv:1708.02967. [Google Scholar]
- Kitazono, J.; Oizumi, M. phi_toolbox.zip, version 6; Figshare. 6 September 2017. Available online: https://figshare.com/articles/phi_toolbox_zip/3203326/6 (accessed on 6 March 2018).
- Barthel, W.; Hartmann, A.K. Clustering analysis of the ground-state structure of the vertex-cover problem. Phys. Rev. E
**2004**, 70, 066120. [Google Scholar] [CrossRef] [PubMed] - Wang, C.; Hyman, J.D.; Percus, A.; Caflisch, R. Parallel tempering for the traveling salesman problem. Int. J. Mod. Phys. C
**2009**, 20, 539–556. [Google Scholar] [CrossRef] - Rathore, N.; Chopra, M.; de Pablo, J.J. Optimal allocation of replicas in parallel tempering simulations. J. Chem. Phys.
**2005**, 122, 024111. [Google Scholar] [CrossRef] [PubMed] - Sugita, Y.; Okamoto, Y. Replica-exchange molecular dynamics method for protein folding. Chem. Phys. Lett.
**1999**, 314, 141–151. [Google Scholar] [CrossRef] - Kofke, D.A. On the acceptance probability of replica-exchange Monte Carlo trials. J. Chem. Phys.
**2002**, 117, 6911–6914, Erratum in**2004**, 120, 10852. [Google Scholar] [CrossRef] - Lee, M.S.; Olson, M.A. Comparison of two adaptive temperature-based replica exchange methods applied to a sharp phase transition of protein unfolding-folding. J. Chem. Phys.
**2011**, 134, 244111. [Google Scholar] [CrossRef] [PubMed] - Gelman, A.; Rubin, D.B. Inference from iterative simulation using multiple sequences. Stat. Sci.
**1992**, 7, 457–472. [Google Scholar] [CrossRef] - Brooks, S.P.; Gelman, A. General methods for monitoring convergence of iterative simulations. J. Comput. Graph. Stat.
**1998**, 7, 434–455. [Google Scholar]

**Figure 1.**Measures of integrated information represented by the Kullback–Leibler divergence between the actual distribution p and q: (

**a**) mutual information; (

**b**) stochastic interaction; and (

**c**) geometric integrated information. The arrows indicate influences across different time points and the lines without arrowheads indicate influences between elements at the same time. This figure is modified from [12].

**Figure 2.**Computational time of the search using Queyranne’s algorithm and the exhaustive search. The red circles and the red solid lines indicate the computational time of the search using Queyranne’s algorithm and their approximate curves ((

**a**) ${log}_{10}T=3.066{log}_{10}N-3.838$, (

**b**) ${log}_{10}T=4.776{log}_{10}N-4.255$). The black triangles and the black dashed lines indicate the computational time of the exhaustive search and their approximate curves ((

**a**) ${log}_{10}T=0.2853N-3.468$, (

**b**) ${log}_{10}T=0.3132N-2.496$).

Model | ${\mathbf{\Phi}}_{\mathbf{SI}}$ | ${\mathbf{\Phi}}_{\mathbf{G}}$ | |||||||
---|---|---|---|---|---|---|---|---|---|

A | $\mathit{\sigma}$ | CR | RA | ER | CORR | CR | RA | ER | CORR |

Normal | 0.01 | 100% | 1 | 0 | 1 | 100% | 1 | 0 | 1 |

0.1 | 100% | 1 | 0 | 1 | 100% | 1 | 0 | 1 | |

Block | 0.01 | 100% | 1 | 0 | 1 | 97% | 1.05 | 2.38 × 10${}^{-3}$ | 0.978 |

0.1 | 100% | 1 | 0 | 1 | 97% | 1.03 | 9.11× 10${}^{-4}$ | 0.978 |

Model | Winning Percentage | Number of Evaluations of $\mathbf{\Phi}$ | |||||
---|---|---|---|---|---|---|---|

A | $\mathit{\sigma}$ | Queyranne’s | Even | REMCMC | Queyranne’s | REMCMC (Mean ± std) | |

Converged | Solution Found | ||||||

Normal | 0.01 | 0% | 100% | 0% | 41,699 | 274,257 ± 107,969 | 8172.6 ± 6291.0 |

0.1 | 0% | 100% | 0% | 41,699 | 315,050 ± 112,205 | 9084.9 ± 7676.4 | |

Block | 0.01 | 0% | 100% | 0% | 41,699 | 308,976 ± 110,905 | 7305.6 ± 6197.0 |

0.1 | 0% | 100% | 0% | 41,699 | 339,869 ± 154,161 | 4533.4 ± 3004.8 |

Model | Winning Percentage | Number of Evaluations of $\mathbf{\Phi}$ | |||||
---|---|---|---|---|---|---|---|

A | $\mathit{\sigma}$ | Queyranne’s | Even | REMCMC | Queyranne’s | REMCMC (Mean ± std) | |

Converged | Solution Found | ||||||

Normal | 0.01 | 0% | 100% | 0% | 2679 | 136,271 ± 46,624 | 862.4 ± 776.3 |

0.1 | 0% | 100% | 0% | 2679 | 122,202 ± 46,795 | 894.3 ± 780.2 | |

Block | 0.01 | 0% | 100% | 0% | 2679 | 129,770 ± 88,483 | 245.2 ± 194.3 |

0.1 | 0% | 100% | 0% | 2679 | 146,034 ± 61,880 | 443.2 ± 642.1 |

${\mathbf{\Phi}}_{\mathbf{SI}}$ | ${\mathbf{\Phi}}_{\mathbf{G}}$ | ||||||
---|---|---|---|---|---|---|---|

CR | RA | ER | CORR | CR | RA | ER | CORR |

100% | 1 | 0 | 1 | 100% | 1 | 0 | 1 |

Winning Percentage | Number of Evaluations of $\mathbf{\Phi}$ | ||||
---|---|---|---|---|---|

Queyranne’s | Even | REMCMC | Queyranne’s | REMCMC (Mean ± std) | |

Converged | Solution Found | ||||

0% | 100% | 0% | 87,423 | 607,797 ± 410,588 | 15,859 ± 10,497 |

© 2018 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Kitazono, J.; Kanai, R.; Oizumi, M. Efficient Algorithms for Searching the Minimum Information Partition in Integrated Information Theory. *Entropy* **2018**, *20*, 173.
https://doi.org/10.3390/e20030173

**AMA Style**

Kitazono J, Kanai R, Oizumi M. Efficient Algorithms for Searching the Minimum Information Partition in Integrated Information Theory. *Entropy*. 2018; 20(3):173.
https://doi.org/10.3390/e20030173

**Chicago/Turabian Style**

Kitazono, Jun, Ryota Kanai, and Masafumi Oizumi. 2018. "Efficient Algorithms for Searching the Minimum Information Partition in Integrated Information Theory" *Entropy* 20, no. 3: 173.
https://doi.org/10.3390/e20030173