Information Geometric Approach on Most Informative Boolean Function Conjecture

Abstract

Let $X^n$ be a memoryless uniform Bernoulli source and $Y^n$ be the output of it through a binary symmetric channel. Courtade and Kumar conjectured that the Boolean function $f:{\{0,1\}}^n\to \{0,1\}$ that maximizes the mutual information $I(f\left(X^n\right);Y^n)$ is a dictator function, i.e., $f\left(x^n\right)=x_i$ for some i. We propose a clustering problem, which is equivalent to the above problem where we emphasize an information geometry aspect of the equivalent problem. Moreover, we define a normalized geometric mean of measures and interesting properties of it. We also show that the conjecture is true when the arithmetic and geometric mean coincide in a specific set of measures.

1. Introduction

Let $X^n$ be an independent and identically distributed (i.i.d.) uniform Bernoulli source and $Y^n$ be an output of it through a memoryless binary symmetric channel with crossover probability $p<1/2$. Recently, Courtade and Kumar conjectured that the most informative Boolean function is a dictator function.

Conjecture 1

([1]). For any Boolean function $f:{\{0,1\}}^n\to \{0,1\}$, we have:

$$\begin{array}{c}\hfill I(f\left(X^n\right);Y^n)\le 1-h_2\left(p\right)\end{array}$$

(1)

where the maximum is achieved by a dictator function, i.e., $f\left(x^n\right)=x_i$ for some $1\le i\le n$. Note that $h_2\left(p\right)=-p\log p-(1-p)\log (1-p)$ is the binary entropy function.

Although there has been some progress in this line of work [2,3], this simple conjecture still remains open. There are also a number of variations of this conjecture. Weinberger and Shayevitz [4] considered the optimal Boolean function under quadratic loss. Huleihel and Ordentlich [5] considered the complementary case and showed that $I(f\left(X^n\right);Y^n)\le (n-1)(1-h_2\left(p\right))$ for all $f:{\{0,1\}}^n\to {\{0,1\}}^{n-1}$. Nazer et al. focused on information distilling quantizers [6], which can be seen as a generalized version of the above problem.

Many of them are based on the Fourier analysis technique including the original paper [1]. In this paper, we suggest an alternative approach, namely the information geometric approach. The mutual information can naturally be expressed with Kullback–Leibler (KL) divergences. Thus, it can be shown that the maximizing mutual information is equivalent to clustering probability measures under KL divergence.

In the equivalent clustering problem, the center of the cluster is an arithmetic mean of measures. We also provide the role of the geometric mean of measures (with appropriate normalization) in this setting. To the best of our knowledge, the geometric mean of measures has received less attention in the literature. We propose an equivalent formulation of the conjecture using the geometric mean of measures. Note that the geometric mean also allows us to connect Conjecture 1 to the other well-known clustering problem.

The rest of the paper is organized as follows. In Section 2, we briefly review the Jensen–Shannon divergence and $\mathcal{I}$-compressedness. In Section 3, we provide an equivalent clustering problem of probability measures. We introduce the geometric mean of measures in Section 4. We conclude this paper in Section 5.

Notations

$\mathcal{X}$ denotes the alphabet set of random variable X, and $\mathcal{M}\left(\mathcal{X}\right)$ denotes the set of measures on $\mathcal{X}$. $X^n$ denotes a random vector $(X_1,X_2,\dots ,X_n)$, while $x^n$ denotes a specific realization of it. If it is clear from the context, $P_{Y|x}$ denotes a conditional distribution of Y given $X=x$, i.e., $P_{Y|x}\left(y\right)=P_{Y|X}\left(y\right|x)$. Similarly, $P_{Y^n|x^n}$ denotes a conditional distribution of $Y^n$ given $X^n=x^n$, i.e., $P_{Y^n|x^n}\left(y^n\right)=P_{Y^n|X^n}\left(y^n\right|x^n)$. Let $\Omega ={\{0,1\}}^n$ be the set of all binary sequences of length n. For $A\subseteq \Omega$, the shifted version of A is denoted by $A\oplus x^n=\{{\tilde{x}}^n\oplus x^n:{\tilde{x}}^n\in A\}$ where ⊕ is an element-wise XOR operator. The arithmetic mean of measures in the set $\{P_{Y^n|x^n}:x^n\in A\}$ is denoted by ${\mu }_A$. For $1\le i\le n$, let $A_{i0}$ be the set of elements in A that satisfy $x_i=0$, i.e., $A_{i0}=\{x^n\in A:x_i=0\}$, and ${\Omega }_{i0}=\{x^n\in \Omega :x_i=0\}$. $A_{i1}$ is defined in a similar manner. A length n binary vector $x^{n-1}0$ denotes a vector $x^n$ with $x_n=0$.

2. Preliminaries

2.1. Jensen–Shannon Divergence

For ${\alpha }_1,{\alpha }_2\ge 0$ such that ${\alpha }_1+{\alpha }_2=1$, the Jensen–Shannon (JS) divergence of two measures $P_1$ and $P_2$ is defined as:

$$\begin{array}{c}\hfill {\mathrm{JSD}}_{\alpha }(P_1,P_2)=H({\alpha }_1{P}_1+{\alpha }_2{P}_2)-{\alpha }_{1}H\left(P_1\right)-{\alpha }_{2}H\left(P_2\right).\end{array}$$

(2)

It is not hard to show that the following definition is equivalent.

$$\begin{array}{c}\hfill {\mathrm{JSD}}_{\alpha }(P_1,P_2)={\alpha }_{1}D(P_1\parallel {\alpha }_1{P}_1+{\alpha }_2{P}_2)+{\alpha }_{2}D(P_2\parallel {\alpha }_1{P}_1+{\alpha }_2{P}_2).\end{array}$$

(3)

Lin proposed a generalized JS divergence [7]:

$$\begin{array}{c}\hfill {\mathrm{JSD}}_{\alpha }(P_1,P_2,\dots ,P_n)=H\left(\sum _{i=1}^n{\alpha }_i{P}_i\right)-\sum _{i=1}^n{\alpha }_{i}H\left(P_i\right)\end{array}$$

(4)

where $\alpha =({\alpha }_1,\dots ,{\alpha }_n)$ is a weight vector such that ${\sum }_{i=1}^n{\alpha }_i=1$. Similar to Equation (3), it has an equivalent definition:

$$\begin{array}{c}\hfill {\mathrm{JSD}}_{\alpha }(P_1,P_2,\dots ,P_n)=\sum _{i=1}^n{\alpha }_{i}D(P_i\parallel \overline{P})\end{array}$$

(5)

where $\overline{P}={\sum }_{i=1}^n{\alpha }_i{P}_i$. Topsøe [8] pointed out an interesting property, the so-called compensation identity. It states that for any distribution Q,

$$\begin{array}{cc}\hfill \sum _{i=1}^n{\alpha }_{i}D(P_i\parallel Q)=& \sum _{i=1}^n{\alpha }_{i}D(P_i\parallel \overline{P})+D(\overline{P}\parallel Q)\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill =& {\mathrm{JSD}}_{\alpha }(P_1,P_2,\dots ,P_n)+D(\overline{P}\parallel Q).\hfill \end{array}$$

(7)

Throughout the paper, we often use Equation (6) directly without the notion of JSD

Remark 1.

The generalized JS divergence is the mutual information between X and the mixture distribution. Let Z be a random variable that takes the value from $\{1,2,\dots ,n\}$ where $P_Z\left(i\right)={\alpha }_i$ and $P_{X|Z}\left(x\right|i)=P_i\left(x\right)$. Then, it is not hard to show that:

$$\begin{array}{c}\hfill {JSD}_{\alpha }(P_1,P_2,\dots ,P_n)=I(X;Z)\end{array}$$

(8)

However, we introduced generalized JS divergence to emphasize the information geometric perspective of our problem.

2.2. $\mathcal{I}$-Compressed

Let A be the subset of $\Omega$ and $\mathcal{I}=\{i_1,i_2,\dots ,i_k\}\subset \{1,2,\dots ,n\}$ be the set of indexes. For $x^n\in \Omega$, the $\mathcal{I}$-section of A is defined as:

$$\begin{array}{c}\hfill A_{\mathcal{I}}\left(x^n\right)=\left\{z^k:y^n\in A,y_i=\left\{\begin{array}{cc}{z}_j\hfill & \mathrm{if}i=i_j\in \mathcal{I}\hfill \\ x_i\hfill & \mathrm{otherwise}\hfill \end{array}\right.\right\}.\end{array}$$

(9)

The set A is called $\mathcal{I}$-compressed if $A_{\mathcal{I}}\left(x^n\right)$ is an initial segment of lexicographical ordering for all $x^n$. For example, if A is $\mathcal{I}$-compressed for some $\left|\mathcal{I}\right|=2$, then $A_{\mathcal{I}}\left(x^n\right)$ should be one of:

$$\begin{array}{c}\hfill \left\{00\right\},\{00,01\},\{00,01,10\},\{00,01,10,11\}.\end{array}$$

(10)

It simply says that if $x^{n-2}10\in A$, then $x^{n-2}00,x^{n-2}01\in A$.

Courtade and Kumar showed that it is enough to consider the $\mathcal{I}$-compressed sets.

Theorem 1

([1]). Let ${\mathcal{S}}_n$ be the set of functions $f:\Omega \to \{0,1\}$ for which $f^{-1}\left(0\right)$ is $\mathcal{I}$-compressed for all $\mathcal{I}$ with $\left|\mathcal{I}\right|\le 2$. In maximizing $I(f\left(X^n\right);Y^n)$, it is sufficient to consider functions $f\in {\mathcal{S}}_n$.

In this paper, we often restrict our attention to functions in the set ${\mathcal{S}}_n$.

3. Approach via Clustering

In this section, we provide an interesting approach toward Conjecture 1 via clustering. More precisely, we formulate an equivalent clustering problem.

3.1. Equivalence to Clustering

The following theorem implies the relation between the original conjecture and the clustering problem.

Theorem 2.

Let $f:\mathcal{X}\to \mathcal{U}$ and $U=f\left(X\right)$ be an induced random variable. Then,

$$\begin{array}{cc}\hfill I\left(f\right(X);Y)=& I(X;Y)-\sum _x{P}_X\left(x\right)D(P_{Y|x}\parallel P_{Y|U}(·|f\left(x\right))).\hfill \end{array}$$

(11)

The proof of the theorem is provided in Appendix A. Note that:

$$\begin{array}{cc}\hfill P_{Y|U}\left(y\right|u)=& \frac{P_{U|Y}\left(u\right|y)P_Y\left(y\right)}{P_U\left(u\right)}\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill =& \frac{P_Y\left(y\right)}{P_U\left(u\right)}\sum _{x\in f^{-1}\left(u\right)}{P}_{X|Y}\left(x\right|y)\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill =& \sum _{x\in f^{-1}\left(u\right)}\frac{P_X\left(x\right)}{P_U\left(u\right)}·P_{Y|X}\left(y\right|x).\hfill \end{array}$$

(14)

which is a weighted mean of $P_{Y|X}\left(y\right|x)$ for $x\in f^{-1}\left(u\right)$. The $D(P_{Y|x}\parallel P_{Y|U}(·|f\left(x\right)))$ is a distance from each element to the cluster center. This implies that maximizing $I\left(f\right(X);Y)$ is equivalent to clustering $\left\{P_{Y^n|x^n}\right\}$ under KL divergences. Since KL divergence is a Bregman divergence, all clusters are separated by a hyperplane [9].

In this paper, we focus on $\mathcal{U}=\{0,1\}$ where $X^n$ is i.i.d. Bern(1/2).

Corollary 1.

Let $f:\Omega \to \{0,1\}$ and $U=f\left(X^n\right)$ be a binary random variable.

$$\begin{array}{cc}\hfill I(f\left(X^n\right);Y^n)=& n-n{h}_2\left(p\right)-\frac{1}{2^n}\sum _{x^n}D(P_{Y^n|x^n}\parallel P_{Y^n|U}(·|f\left(x^n\right))).\hfill \end{array}$$

(15)

The equivalent clustering problem is minimizing:

$$\begin{array}{c}\hfill \sum _{x^n}D(P_{Y^n|x^n}\parallel P_{Y^n|U}(·|f\left(x^n\right))).\end{array}$$

(16)

Let $A=\{x^n\in \Omega :f\left(x^n\right)=0\}$, then we can simplify $P_{Y^n|U}$ further.

$$\begin{array}{cc}\hfill P_{Y^n|U}\left(y^n\right|0)=& \frac{1}{\left|A\right|}\sum _{x^n\in A}{P}_{Y^n|X^n}\left(y^n\right|x^n)\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill \stackrel{\Delta }{=}& {\mu }_A\left(y^n\right).\hfill \end{array}$$

(18)

The cluster center ${\mu }_A$ is an arithmetic mean of measures in the set $\{P_{Y^n|x^n}:x^n\in A\}$. Then, we have:

$$\begin{array}{cc}\hfill \sum _{x^n}D(P_{Y^n|x^n}\parallel P_{Y^n|U}(·|f\left(x^n\right)))=& \sum _{x^n\in A}D(P_{Y^n|x^n}\parallel {\mu }_A)+\sum _{x^n\in A^c}D(P_{Y^n|x^n}\parallel {\mu }_{A^c}).\hfill \end{array}$$

(19)

For simplicity, let:

$$\begin{array}{c}\hfill \mathcal{D}\left(A\right)\stackrel{\Delta }{=}\sum _{x^n\in A}D(P_{Y^n|x^n}\left|\right|{\mu }_A)\end{array}$$

(20)

which is the sum of distances from each element in A to the cluster center. In short, finding the most informative Boolean function f is equivalent to finding the set $A\subseteq \Omega$ that minimizes $\mathcal{D}\left(A\right)+\mathcal{D}\left(A^c\right)$.

Remark 2.

Conjecture 1 implies that $A={\Omega }_{i0}=\{x^n:x_i=0\}$ minimizes (19). Furthermore, Theorem 1 implies that it is enough to consider A such that $A_{i1}\subseteq A_{i0}$ for all i.

For any $Q_{Y^n}\in \mathcal{M}(\Omega )$, Equation (6) implies that:

$$\begin{array}{c}\hfill \sum _{x\in A}D(P_{Y^n|x^n}\parallel Q_{Y^n})=\mathcal{D}\left(A\right)+\left|A\right|D({\mu }_A\parallel Q_{Y^n})\end{array}$$

(21)

$$\begin{array}{c}\hfill \sum _{x\in A^c}D(P_{Y^n|x^n}\parallel Q_{Y^n})=\mathcal{D}\left(A^c\right)+\left|A^c\right|D({\mu }_{A^c}\parallel Q_{Y^n}).\end{array}$$

(22)

Thus, we have:

$$\begin{array}{c}\hfill \sum _{x\in \Omega }D(P_{Y^n|x^n}\parallel Q_{Y^n})=\mathcal{D}\left(A\right)+\mathcal{D}\left(A^c\right)+\left|A\right|D({\mu }_A\parallel Q_{Y^n})+\left|A^c\right|D({\mu }_{A^c}\parallel Q_{Y^n}).\end{array}$$

(23)

Note that ${\sum }_{x\in \Omega }D(P_{Y^n|x^n}\parallel Q_{Y^n})$ does not depend on A, and therefore, we have the following theorem.

Theorem 3.

For any $Q_{Y^n}\in \mathcal{M}(\Omega )$, minimizing $\mathcal{D}\left(A\right)+\mathcal{D}\left(A^c\right)$ is equivalent to maximizing:

$$\begin{array}{c}\hfill \left|A\right|D({\mu }_A\parallel Q_{Y^n})+\left|A^c\right|D({\mu }_{A^c}\parallel Q_{Y^n}).\end{array}$$

(24)

The above theorem provides an alternative problem formulation of the original conjecture.

3.2. Connection to Clustering under Hamming Distance

In this section, we consider the duality between the above clustering problem under the KL divergence and the clustering on $\Omega$ under the Hamming distance. The following theorem shows that the KL divergence on $\{P_{Y^n|x^n}:x^n\in \Omega \}$ corresponds to the Hamming distance on $\Omega$.

Theorem 4.

For all $x^n,{\tilde{x}}^n\in \Omega$, we have:

$$\begin{array}{c}\hfill D(P_{Y^n|x^n}\parallel P_{Y^n{|}^{{\tilde{x}}^n}})=d_H(x^n,{\tilde{x}}^n)·(1-2p)\log \frac{1-p}{p}\end{array}$$

(25)

where $d_H(x^n,{\tilde{x}}^n)$ denotes the Hamming distance between $x^n$ and ${\tilde{x}}^n$.

This theorem implies that the distance between two measures $P_{Y^n|x^n}$ and $P_{Y^n|{\tilde{x}}^n}$ is proportional to the Hamming distance between two binary vectors $x^n$ and ${\tilde{x}}^n$. The proof of the theorem is provided in Appendix B. Note that the KL divergence $D(·\parallel ·)$ is symmetric on $\{P_{Y^n|x^n}:x^n\in \Omega \}$.

In the above duality, we have a mapping between $\{P_{Y^n|x^n}:x^n\in \Omega \}$ and ${\{0,1\}}^n$; more precisely, $P_{Y^n|x^n}\leftrightarrow x^n$. This mapping naturally suggests an equivalent clustering problem of n-dimensional binary vectors. However, the cluster center ${\mu }_A$ is not an element of $\{P_{Y^n|x^n}:x^n\in \Omega \}$ in general. In order to formulate an equivalent clustering problem, we need to answer the question “Which n dimensional vector corresponds to ${\mu }_A$?”. A naive approach is to extend the set of binary vectors to ${[0,1]}^n$ under ${\ell }^2$ distance instead of the Hamming distance. In such a case, the goal is to map ${\mu }_A$ to the arithmetic mean of binary vectors in the set A. If this is true, we can further simplify the problem into the problem of clustering a hypercube in ${\mathbb{R}}^n$. However, the following example shows that this naive extension is not valid.

Example 1.

Let $n=2$, $A=\{00,11\}$ and $B=\{01,10\}$, then the arithmetic mean of binary vectors of A and that of B are the same. However, ${\mu }_A$ is not equal to ${\mu }_B$.

Furthermore, the set ${\Omega }_{i0}$ is not the optimum choice when clustering the hypercube under ${\ell }^2$. Instead, we need to consider the set of measures directly. The following theorem provides a bit of geometric structure among measures.

Theorem 5.

For all $x^n,{\tilde{x}}^n\in \Omega$ and $Q_{Y^n}\in \left(\left\{P_{Y^n|x^n}\right|x^n\in \Omega \}\right)$,

$$\begin{array}{cc}\hfill D(P_{Y^n|X^n=x^n}\parallel Q_{Y^n})-D(P_{Y^n|X^n={\tilde{x}}^n}\parallel Q_{Y^n})\le & k·({(1-p)}^k-p^k)\log \frac{1-p}{p}\hfill \end{array}$$

(26)

where $k=d_H(x^n,{\tilde{x}}^n)$.

The proof of the theorem is provided in Appendix C. Since ${(1-p)}^k-p^k\le 1-2p$ for all $k\ge 1$, Theorem 5 immediately implies the following corollary.

Corollary 2.

For all $x^n,{\tilde{x}}^n\in \Omega$ and $Q_{Y^n}\in \mathrm{conv}\left(\left\{P_{Y^n|x^n}\right|x^n\in \Omega \}\right)$,

$$\begin{array}{c}\hfill \left|D(P_{Y^n|X^n=x^n}\parallel Q_{Y^n})-D(P_{Y^n|X^n={\tilde{x}}^n}\parallel Q_{Y^n})\right|\le D(P_{Y^n|X^n=x^n}\parallel P_{Y^n|X^n={\tilde{x}}^n})\end{array}$$

(27)

where $\mathrm{conv}\left(A\right)$ is a convex hull of measures in the set A.

This is a triangle inequality that can be useful when we consider the clustering problem of measures.

4. Geometric Mean of Measures

In the previous section, we formulate the clustering problem that is equivalent to the original maximizing mutual information problem. In this section, we provide another approach using a geometric mean of measures. We define the geometric mean of measures formally and derive a nontrivial conjecture, which is equivalent to Conjecture 1.

4.1. Definition of the Geometric Mean of Measures

For measures $P_1,P_2,\dots ,P_n\in \mathcal{M}\left(\mathcal{X}\right)$ and weights ${\alpha }_i\ge 0$ such that ${\sum }_{i=1}^n{\alpha }_i=1$, we considered the sum of KL divergences in (6):

$$\begin{array}{c}\hfill \sum _{i=1}^n{\alpha }_{i}D(P_i\parallel Q).\end{array}$$

(28)

We also observed that (28) is minimized when Q is an arithmetic mean of measures.

Since the KL divergence is asymmetric, it is natural to consider the sum of another direction of KL divergences.

$$\begin{array}{cc}\hfill \sum _{i=1}^n{\alpha }_{i}D(Q\parallel P_i)=& \sum _{i=1}^n{\alpha }_i\sum _{x\in \mathcal{X}}Q\left(x\right)\log \frac{Q\left(x\right)}{P_i\left(x\right)}\hfill \end{array}$$

(29)

$$\begin{array}{cc}\hfill =& \sum _{x\in \mathcal{X}}\sum _{i=1}^n{\alpha }_{i}Q\left(x\right)\log \frac{Q\left(x\right)}{P_i\left(x\right)}\hfill \end{array}$$

(30)

$$\begin{array}{cc}\hfill =& \sum _{x\in \mathcal{X}}Q\left(x\right)\log \frac{Q\left(x\right)}{{\prod }_{i=1}^n{\left(P_i\left(x\right)\right)}^{{\alpha }_i}}.\hfill \end{array}$$

(31)

Compared to the arithmetic mean that minimizes (28), ${\prod }_{i=1}^n{\left(P_i\left(x\right)\right)}^{{\alpha }_i}$ can be considered as a geometric mean of measures. However, ${\prod }_{i=1}^n{\left(P_i\left(x\right)\right)}^{{\alpha }_i}$ is not a measure in general, and normalization is required. With a normalizing constant s, we can define the geometric mean of measures by:

$$\begin{array}{c}\hfill {\overline{P}}_G\left(x\right)=\frac{1}{s}\prod _{i=1}^n{\left(P_i\left(x\right)\right)}^{{\alpha }_i}\end{array}$$

(32)

where s is a constant, so that ${\sum }_{x\in \mathcal{X}}{\overline{P}}_G\left(x\right)=1$, i.e.,

$$\begin{array}{c}\hfill s=\sum _x\prod _{i=1}^n{\left(P_i\left(x\right)\right)}^{{\alpha }_i}.\end{array}$$

(33)

Then, we have:

$$\sum _{i=1}^n{\alpha }_{i}D(Q\parallel P_i)=D(Q\parallel {\overline{P}}_G)+\log \frac{1}{s}$$

(34)

which can be minimized when $Q={\overline{P}}_G$. Thus, for all Q,

$$\begin{array}{cc}\hfill \sum _{i=1}^n{\alpha }_{i}D(Q\parallel P_i)\ge & \sum _{i=1}^n{\alpha }_{i}D({\overline{P}}_G\parallel P_i)\hfill \end{array}$$

(35)

$$=\log \frac{1}{s}.$$

(36)

The above result provides a geometric compensation identity.

$$\begin{array}{cc}\hfill \sum _{i=1}^n{\alpha }_{i}D(Q\parallel P_i)=& D(Q\parallel {\overline{P}}_G)+\sum _{i=1}^n{\alpha }_{i}D({\overline{P}}_G\parallel P_i).\hfill \end{array}$$

(37)

This also implies that $\log \frac{1}{s}\ge 0$.

Remark 3.

If $n=2$, s is called the α-Chernoff coefficient, and it is called the Bhattacharyya coefficient when $\alpha =1/2$. The summation $\log \frac{1}{s}={\sum }_{i=1}^2{\alpha }_{i}D({\overline{P}}_G\parallel P_i)$ is known as α-Chernoff divergence. For more details, please see [10,11] and the references therein.

Under this definition, we can find the geometric mean of measures in the set $\{P_{Y^n|{\tilde{x}}^n}:{\tilde{x}}^n\in B\}$ with uniform weights $\frac{1}{\left|B\right|}$ by:

$$\begin{array}{c}\hfill {\gamma }_B\left(y^n\right)=\frac{1}{s_B}{\left(\prod _{{\tilde{x}}^n\in B}{P}_{Y^n|X^n}\left(y^n\right|{\tilde{x}}^n)\right)}^{1/\left|B\right|}\end{array}$$

(38)

where:

$$\begin{array}{c}\hfill s_B=\sum _{y^n}{\left(\prod _{{\tilde{x}}^n\in B}{P}_{Y^n|X^n}\left(y^n\right|{\tilde{x}}^n)\right)}^{1/\left|B\right|}.\end{array}$$

(39)

Remark 4.

The original conjecture is that the Boolean function f such that $f^{-1}\left(0\right)={\Omega }_{i0}=\{x^n:x_i=0\}$ maximizes the mutual information $I(f\left(X^n\right);Y^n)$. The geometric mean of measures in the set $\{P_{Y^n|x^n}:x^n\in {\Omega }_{i0}\}$ satisfies the following property.

$$\begin{array}{cc}\hfill {\gamma }_{{\Omega }_{i0}}=& {\mu }_{{\Omega }_{i0}}\hfill \end{array}$$

(40)

$$\begin{array}{cc}\hfill s_{{\Omega }_{i0}}=& 2^{n-1}{\left(p(1-p)\right)}^{(n-1)/2}.\hfill \end{array}$$

(41)

Note that the geometric mean of measures in the set $\{P_{Y^n|x^n}:x^n\in \Omega \}$ satisfies:

$${\gamma }_{\Omega }={\mu }_{\Omega }$$

(42)

$$s_{\Omega }=2^n{\left(p(1-p)\right)}^{n/2}.$$

(43)

4.2. Main Results

So far, we have seen two means of measures ${\mu }_A$ and ${\gamma }_A$. It is natural to ask if they are equal. Our main theorem provides a connection to Conjecture 1.

Theorem 6.

Suppose A is a nontrivial subset of $\Omega ={\{0,1\}}^n$ for $n>0$ (i.e., $A\ne \varnothing ,\Omega$), and A is $\mathcal{I}$-compressed for all $\left|\mathcal{I}\right|=1$. Then, $A={\Omega }_{i0}$ for some i if and only if ${\mu }_A={\gamma }_A$ and ${\mu }_{A^c}={\gamma }_{A^c}$.

The proof of the theorem is provided in Appendix D. Theorem 6 implies that the following conjecture is the equivalent to Conjecture 1.

Conjecture 2.

Let $f:\Omega \to \{0,1\}$ and $A=f^{-1}\left(0\right)$ be $\mathcal{I}$-compressed for all $\left|\mathcal{I}\right|=1$. Then, $I\left(f\right(X);Y)$ is maximized if and only if ${\mu }_A={\gamma }_A$ and ${\mu }_{A^c}={\gamma }_{A^c}$.

Remark 5.

One of the main challenges of this problem is that the conjectured optimal sets are extremes, i.e., $A={\Omega }_{i0}$ for some i. Our main theorem provides an alternative conjecture that seems more natural in the context of optimization.

Remark 6.

It is clear that ${\mu }_A={\gamma }_A$ holds if $\left|A\right|=1$. Thus, both conditions ${\mu }_A={\gamma }_A$ and ${\mu }_{A^c}={\gamma }_{A^c}$ are needed to guarantee $A={\Omega }_{i0}$ for some i.

4.3. Property of the Geometric Mean

We can derive a new identity by combining the original and geometric compensation identity together. For $A,B\subset \Omega$, let $\pi (A,B)$ be:

$$\begin{array}{c}\hfill \pi (A,B)=\sum _{(x^n,{\tilde{x}}^n)\in A\times B}D(P_{Y^n|x^n}\parallel P_{Y^n|{\tilde{x}}^n}).\end{array}$$

(44)

Then,

$$\begin{array}{cc}\hfill \pi (A,B)=& \sum _{(x^n,{\tilde{x}}^n)\in A\times B}D(P_{Y^n|x^n}\parallel P_{Y^n|{\tilde{x}}^n})\hfill \end{array}$$

(45)

$$\begin{array}{cc}\hfill =& \sum _{{\tilde{x}}^n\in B}\sum _{x^n\in A}D(P_{Y^n|x^n}\parallel P_{Y^n|{\tilde{x}}^n})\hfill \end{array}$$

(46)

$$\begin{array}{cc}\hfill =& \sum _{{\tilde{x}}^n\in B}\left(\sum _{x^n\in A}D(P_{Y^n|x^n}\parallel {\mu }_A)+\left|A\right|D({\mu }_A\parallel P_{Y|{\tilde{x}}^n})\right)\hfill \end{array}$$

(47)

$$\begin{array}{cc}\hfill =& \left|B\right|\mathcal{D}\left(A\right)+\left|A\right|\sum _{{\tilde{x}}^n\in B}D({\mu }_A\parallel P_{Y|{\tilde{x}}^n})\hfill \end{array}$$

(48)

where (47) is because of the compensation identity (6). As we discussed in Section 4.1, the second term of the right-hand side is:

$$\begin{array}{cc}\hfill \sum _{{\tilde{x}}^n\in B}D({\mu }_A\parallel P_{{\tilde{x}}^n})=& \sum _{{\tilde{x}}^n\in B}\sum _{y^n}{\mu }_A\left(y^n\right)\log \frac{{\mu }_A\left(y^n\right)}{P_{Y^n|X^n}\left(y^n\right|{\tilde{x}}^n)}\hfill \end{array}$$

(49)

$$\begin{array}{cc}\hfill =& \left|B\right|\sum _{y^n}{\mu }_A\left(y^n\right)\log \frac{{\mu }_A\left(y^n\right)}{{\left({\prod }_{{\tilde{x}}^n\in B}{P}_{Y^n|X^n}\left(y^n\right|{\tilde{x}}^n)\right)}^{1/\left|B\right|}}\hfill \end{array}$$

(50)

$$\begin{array}{cc}\hfill =& \left|B\right|D({\mu }_A\parallel {\gamma }_B)+\left|B\right|\log \frac{1}{s_B}\hfill \end{array}$$

(51)

$$\begin{array}{cc}\hfill =& \left|B\right|D({\mu }_A\parallel {\gamma }_B)+\sum _{{\tilde{x}}^n\in B}D({\gamma }_B\parallel P_{Y^n|{\tilde{x}}^n}).\hfill \end{array}$$

(52)

Finally, we have:

$$\begin{array}{cc}\hfill \frac{1}{\left|A\right|\left|B\right|}\pi (A,B)=& \frac{1}{\left|A\right|}\mathcal{D}\left(A\right)+D({\mu }_A\parallel {\gamma }_B)+\log \frac{1}{s_B}\hfill \end{array}$$

(53)

$$\begin{array}{cc}\hfill =& \frac{1}{\left|A\right|}\sum _{x^n\in A}D(P_{Y^n|x^n}\parallel {\mu }_A)+D({\mu }_A\parallel {\gamma }_B)+\frac{1}{\left|B\right|}\sum _{{\tilde{x}}^n\in B}D({\gamma }_B\parallel P_{Y^n|{\tilde{x}}^n}).\hfill \end{array}$$

(54)

More interestingly, we can apply original and geometric compensation identities:

$$\begin{array}{cc}\hfill \frac{1}{\left|A\right|\left|B\right|}\pi (A,B)=& \frac{1}{\left|A\right|}\sum _{x^n\in A}D(P_{Y^n|x^n}\parallel {\mu }_A)+\frac{1}{\left|B\right|}\sum _{{\tilde{x}}^n\in B}D({\mu }_A\parallel P_{Y^n|{\tilde{x}}^n})\hfill \end{array}$$

(55)

$$\begin{array}{cc}\hfill =& \frac{1}{\left|A\right|}\sum _{x^n\in A}D(P_{Y^n|x^n}\parallel {\gamma }_B)+\frac{1}{\left|B\right|}\sum _{{\tilde{x}}^n\in B}D({\gamma }_B\parallel P_{Y^n|{\tilde{x}}^n}).\hfill \end{array}$$

(56)

From Theorem 4, we have $\pi (A,B)=\pi (B,A)$, and therefore, we can switch A and B.

$$\begin{array}{cc}\hfill \frac{1}{\left|A\right|\left|B\right|}\pi (A,B)=& \frac{1}{\left|B\right|}\sum _{x^n\in B}D(P_{Y^n|x^n}\parallel {\mu }_B)+D({\mu }_B\parallel {\gamma }_A)+\frac{1}{\left|A\right|}\sum _{{\tilde{x}}^n\in A}D({\gamma }_A\parallel P_{Y^n|{\tilde{x}}^n})\hfill \end{array}$$

(57)

$$\begin{array}{cc}\hfill =& \frac{1}{\left|B\right|}\sum _{x^n\in B}D(P_{Y^n|x^n}\parallel {\mu }_B)+\frac{1}{\left|A\right|}\sum _{{\tilde{x}}^n\in A}D({\mu }_B\parallel P_{Y^n|{\tilde{x}}^n})\hfill \end{array}$$

(58)

$$\begin{array}{cc}\hfill =& \frac{1}{\left|B\right|}\sum _{x^n\in B}D(P_{Y^n|x^n}\parallel {\gamma }_A)+\frac{1}{\left|A\right|}\sum _{{\tilde{x}}^n\in A}D({\gamma }_A\parallel P_{Y^n|{\tilde{x}}^n}).\hfill \end{array}$$

(59)

If we let $A=B$, we have:

$$\begin{array}{cc}\hfill \frac{1}{{\left|A\right|}^2}\pi (A,A)=& \frac{1}{\left|A\right|}\sum _{x^n\in A}\left[D(P_{Y^n|x^n}\parallel {\gamma }_A)+D({\gamma }_A\parallel P_{Y^n|x^n})\right]\hfill \end{array}$$

(60)

$$\begin{array}{cc}\hfill =& \frac{1}{\left|A\right|}\sum _{x^n\in A}\left[D(P_{Y^n|x^n}\parallel {\mu }_A)+D({\mu }_A\parallel P_{Y^n|x^n})\right]\hfill \end{array}$$

(61)

$$\begin{array}{cc}\hfill =& \frac{1}{\left|A\right|}\sum _{x^n\in A}\left[D(P_{Y^n|x^n}\parallel {\mu }_A)+D({\mu }_A\parallel {\gamma }_A)+D({\gamma }_A\parallel P_{Y^n|x^n})\right].\hfill \end{array}$$

(62)

Note that $\frac{1}{\left|A\right|}\pi (A,A)+\frac{1}{|A^c|}\pi (A^c,A^c)$ is similar to a known clustering problem. In the clustering literature, the min-sum clustering problem [12] is minimizing the sum of all edges in each cluster. Using $\pi$, we can describe the binary min-sum clustering problem on $\Omega$ by minimizing $\pi (A,A)+\pi (A^c,A^c)$.

4.4. Another Application of the Geometric Mean

Using the geometric mean of measures, we can rewrite the clustering problem in a different form. Recall that ${\mu }_{A\oplus x^n}=\{{\tilde{x}}^n\oplus x^n:{\tilde{x}}^n\in A\}$. Then, we have:

$$\begin{array}{cc}\hfill \mathcal{D}\left(A\right)=& \sum _{x^n\in A}D(P_{Y^n|x^n}\parallel {\mu }_A)\hfill \end{array}$$

(63)

$$\begin{array}{cc}\hfill =& \sum _{x^n\in A}D(P_{Y^n|0^n}\parallel {\mu }_{A\oplus x^n}).\hfill \end{array}$$

(64)

Let ${\tilde{\gamma }}_A$ be the geometric mean of measures in the set $\{{\mu }_{A\oplus x^n}:x^n\in A\}$, i.e.,

$$\begin{array}{c}\hfill {\tilde{\gamma }}_A\left(y^n\right)=\frac{1}{{\tilde{s}}_A}{\left(\prod _{x^n\in A}{\mu }_{A\oplus x^n}\left(y^n\right)\right)}^{1/\left|A\right|}\end{array}$$

(65)

where:

$$\begin{array}{c}\hfill {\tilde{s}}_A=\sum _{y^n}{\left(\prod _{x^n\in A}{\mu }_{A\oplus x^n}\left(y^n\right)\right)}^{1/\left|A\right|}.\end{array}$$

(66)

Then, we have:

$$\begin{array}{c}\hfill \mathcal{D}\left(A\right)=\left|A\right|D(P_{Y^n|0^n}\parallel {\tilde{\gamma }}_A)+\left|A\right|\log \frac{1}{{\tilde{s}}_A}.\end{array}$$

(67)

Needless to say:

$$\begin{array}{c}\hfill \mathcal{D}\left(A^c\right)=|A^c|D(P_{Y^n|0^n}\parallel {\tilde{\gamma }}_{A^c})+\left|A^c\right|\log \frac{1}{{\tilde{s}}_{A^c}}.\end{array}$$

(68)

The sum of the results is:

$$\begin{array}{cc}\hfill \mathcal{D}\left(A\right)+\mathcal{D}\left(A^c\right)=& \left|A\right|D(P_{Y^n|0^n}\parallel {\tilde{\gamma }}_A)+|A^c|D(P_{Y^n|0^n}\parallel {\tilde{\gamma }}_{A^c})+\left|A\right|\log \frac{1}{{\tilde{s}}_A}+\left|A^c\right|\log \frac{1}{{\tilde{s}}_{A^c}}.\hfill \end{array}$$

(69)

This can be considered as a dual of Theorem 3.

Remark 7.

Let ${\Omega }_{i0}=\{x^n:x_i=0\}$, which is the candidate of the optimizer. Then,

$$\begin{array}{cc}\hfill {\tilde{\gamma }}_{{\Omega }_{i0}}=& {\tilde{\gamma }}_{{\left({\Omega }_{i0}\right)}^c}={\mu }_{{\Omega }_{i0}}\hfill \end{array}$$

(70)

$$\begin{array}{cc}\hfill {\tilde{s}}_{{\Omega }_{i0}}=& {\tilde{s}}_{{\left({\Omega }_{i0}\right)}^c}=1.\hfill \end{array}$$

(71)

5. Concluding Remarks

In this paper, we have proposed a number of different formulations of the most informative Boolean function conjecture. Most of them are based on the information geometric approach. Furthermore, we focused on the (normalized) geometric mean of measures that can simplify the problem formulation. More precisely, we showed that Conjecture 1 is true if and only if the maximum achieving f satisfies the following property: “the arithmetic and geometric mean of measures are the same for both $\{P_{Y^n|x^n}:x^n\in f^{-1}\left(0\right)\}$, as well as $\{P_{Y^n|x^n}:x^n\in f^{-1}\left(1\right)\}$.”