An Information-Theoretic Proof of a Hypercontractive Inequality

Abstract

The famous hypercontractive estimate discovered independently by Gross, Bonami and Beckner has had a great impact on combinatorics and theoretical computer science since it was first used in this setting in a seminal paper by Kahn, Kalai and Linial. The usual proofs of this inequality begin with two-point space, where some elementary calculus is used and then generalised immediately by introducing another dimension using submultiplicativity (Minkowski’s integral inequality). In this paper, we prove this inequality using information theory. We compare the entropy of a pair of correlated vectors in ${\{0,1\}}^n$ to their separate entropies, analysing them bit by bit (not as a figure of speech, but as the bits are revealed) using the chain rule of entropy.

1. Introduction

The inequality that we consider in this note is a two-function version of a famous hypercontractive inequality discovered, independently, by Gross [1], Bonami [2] and Beckner [3]. This inequality, first introduced to the combinatorial landscape in a seminal paper [4] by Kahn, Kalai and Linial (henceforth KKL), has become one of the cornerstones of the analytical approach to Boolean functions and theoretical computer science; see, e.g., [5,6,7,8,9,10], and many others. See chapter 16 of [11] for a historical background.

Let $ϵ\in (0,1)$. We will be considering an operator $T_{ϵ}$, which acts on real-valued functions in ${\{0,1\}}^n$. We consider two equivalent definitions of the operator: a spectral definition and a more probabilistic one. The first definition is defined via the eigenfunctions and eigenvalues of the operator. The eigenfunctions are precisely the Walsh–Fourier variables, $\left\{u_x\right\}:x\in {\{0,1\}}^n$, which form a complete orthonormal system under the inner product $<f,g>:=E\left[f\right(x\left)g\right(x\left)\right]$, where the expectation of a function is defined by $E\left[h\left(x\right)\right]=2^{-n}{\sum }_{x}h\left(x\right)$. We recall the definition of these variables. For $x,y\in {\{0,1\}}^n$,

$$u_x\left(y\right)={(-1)}^{{\sum }_i{x}_i{y}_i}.$$

Given a function $f:{\{0,1\}}^n\to \mathbb{R}$ and its unique Fourier expansion, $f={\sum }_x\widehat{f}\left(x\right)u_x$, the action of $T_{ϵ}$ on f is defined by

$$T_{ϵ}\left(f\right)=\sum _x{ϵ}^{{\sum }_i{x}_i}\widehat{f}\left(x\right)u_x.$$

This definition of $T_{ϵ}$ stresses the fact that it “focuses” on the low-frequency part of the Fourier spectrum, an idea that was a crucial element in the KKL proof [4].

For the other definition, let X and Y be random variables taking values in ${\{0,1\}}^n$. For any distribution of X, let Y be such that for every coordinate i, independently, $Y_i$ is chosen so that $Pr[X_i=Y_i]={\displaystyle \frac{1+ϵ}{2}}$. (Or, if one prefers the ${\{-1,1\}}^n$ setting, the restriction is $E\left[X_i{Y}_i\right]=ϵ$.) Note that if X is chosen uniformly, then the marginal distribution of Y is also uniform. We call such a pair $(X,Y)$ an $ϵ$-correlated pair. Then, one can define

$$T_{ϵ}\left(f\right)\left(x\right)=E\left[f\left(Y\right)\right|X=x].$$

It is not hard to verify that these two definitions of $T_{ϵ}$ are equivalent. The second definition, which is the one we will be working with in this paper, stresses the connection of this operator to random walks and isoperimetric inequalities, as it enables one to bind the probability of an $ϵ$-correlated pair of random points $X,Y$ to belong to given sets. It also explains the fact (which will be made formal shortly) that $T_{ϵ}\left(f\right)$ is smoother than f, as the operator is an averaging operator. Without further ado, here is the statement of the inequality:

Theorem 1

([1,2,3]). Let $f:{\{0,1\}}^n\to {\mathbb{R}}_{\ge 0}$ and let $ϵ\in [0,1]$. Then,

$$|T_{ϵ}{\left(f\right)|}_2\le {\left|f\right|}_{1+{ϵ}^2},$$

(1)

where ${\left|g\right|}_p:={\left(E\right[\left|g\right|}^p{\left]\right)}^{1/p}$ and the expectation is written with respect to a uniform measure.

Remarks:

• There are various refinements of this inequality, either dealing with non-uniform measures, the norm of $T_{ϵ}$ as an operator ranging from $L^p$ to $L^q$ for $q\ne 2$, products of a base space with more than two points, or a reverse inequality that deals with the case $p,q<1$; see [10,12,13,14]. It would not be surprising if the method used in this note could be extended to cover such cases too.
• It is not difficult to see that (1) is equivalent to the following: Let $f,g:{\{0,1\}}^n\to \mathbb{R}$; let X be uniformly distributed on ${\{0,1\}}^n$; and let $X,Y$ be an $ϵ$-correlated pair. Then,

  $$E\left[f\left(X\right)g\left(Y\right)\right]\le {\left|f\right|}_{1+ϵ}{\left|g\right|}_{1+ϵ}.$$

  (2)
  This is the inequality proved in this paper.
• A major portion of the hypercontractive inequality’s applications deal with the case where f and g are Boolean functions. We will start our proof with this setting and then show how a small variation deals with the general case.

In this paper, we apply an information-theoretic approach to proving (2) for Boolean functions, trying to analyse the pair $(X,Y)$ as the coordinates of X and Y are revealed to us one by one. Since all known direct proofs of (1) use induction, it is not surprising that one should adopt such a sequential approach. The difference is that the usual proofs begin with two-point space and proceed through induction using the submultiplicativity of the product operator and Minkowski’s integral inequality, whereas we use the chain rule of entropy, exposing the bits of the vectors in question one by one and comparing the amount of information about their joint distribution with the information captured by their marginal distributions. Fortunately, it turns out that regardless of the prefixes revealed so far, at every step, the conditional entropies obey the same inequality.

This is not the first application of entropy to this hypercontractive inequality. The connection between hypercontractivity-type inequalities and information theory was perhaps first addressed in [15]. In [16], the dual form of the hypercontractive inequality is proven for the case of comparing the 2-norm and the 4-norm of a low-degree polynomial acting on ${\{0,1\}}^n$. Blais and Tan, [17], managed to improve this approach and, surprisingly, extract the precise optimal hypercontractive constant for comparing the 2-norm and the q-norm of such polynomials for all positive even integers q. Both these proofs analyse the Fourier space rather than the primal space and use no induction at all.

I will make one final remark regarding the proof in this paper. Although it is not difficult, it is probably, to date, the most involved proof of the inequality from a technical point of view. I believe that it is nonetheless worthwhile to add it to the list of existing proofs because it offers a new point of view which directly addresses the notion of studying the projections of the joint distribution of a pair of $ϵ$-correlated vectors.

2. Main Theorem

2.1. The Boolean Case

Theorem 2.

Let $ϵ\in [0,1]$, and let $\mathcal{X},\mathcal{Y}\subseteq {\{0,1\}}^n$ be nonempty. Let X be uniformly distributed in ${\{0,1\}}^n$, and let Y be such that for each $1\le i\le n$, independently, $Pr[X_i=Y_i]={\displaystyle \frac{1+ϵ}{2}}$. Then,

$$E\left[{\mathbf{1}}_{\mathcal{X}}\left(X\right){\mathbf{1}}_{\mathcal{Y}}\left(Y\right)\right]\le {\left(\mu \left(\mathcal{X}\right)\mu \left(\mathcal{Y}\right)\right)}^{{\displaystyle \frac{1}{1+ϵ}}}$$

(3)

where μ denotes a uniform measure.

Proof. 

For $x,y\in {\{0,1\}}^n$, let $a(x,y)$ denote the number of coordinates on which x and y agree and $d(x,y)$ be the number of coordinates on which they differ. Then, the theorem is equivalent (by straightforward manipulation) to

$$log\left(\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}{(1+ϵ)}^{a(x,y)}{(1-ϵ)}^{d(x,y)}\right)\le {\displaystyle \frac{1}{1+ϵ}}\left(2ϵn+log\left(\right|\mathcal{X}\left|\right)+log\left(\right|\mathcal{Y}\left|\right)\right),$$

(4)

where all logs are base 2. As is usual in proofs using entropy, it suffices, through continuity, to treat the case where $ϵ$ is rational. Let $s\le r$ be natural numbers such that ${\displaystyle \frac{1+ϵ}{2}}={\displaystyle \frac{r}{r+s}}$ and ${\displaystyle \frac{1-ϵ}{2}}={\displaystyle \frac{s}{r+s}}$. Then, (2) reduces to

$$log\left(\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}{r}^{a(x,y)}{s}^{d(x,y)}\right)\le n(log(r+s)-s/r)+{\displaystyle \frac{r+s}{2r}}(log(\left|\mathcal{X}\right|)+log\left(\right|\mathcal{Y}\left|\right)).$$

(5)

We will express the left-hand side of this expression as the entropy of a random variable and proceed to expand it according to the chain rule. First, let $A_{00},A_{11},A_{10}$ and $A_{01}$ be four disjoint sets with

$$\left|A_{00}\right|=\left|A_{11}\right|=r, \left|A_{01}\right|=\left|A_{10}\right|=s.$$

Next, let $(X,Y,Z)$ be a random variable which is distributed uniformly over all triples such that $X\in \mathcal{X},Y\in \mathcal{Y},$ and for every $1\le i\le n$, $Z_i\in A_{X_i{Y}_i}.$ Clearly, Z determines X and Y so that

$$H(X,Y,Z)=H\left(Z\right)=log\left(\sum _{x\in \mathcal{X}}\sum _{y\in \mathcal{Y}}{r}^{a(x,y)}{s}^{d(x,y)}\right).$$

Next, for any vector $w\in {\{0,1\}}^n$, we denote $(w_1,\dots ,w_{i-1}):=w_{<i}$. So, from the chain rule, we have

$$H\left(Z\right)=\sum _{i}H\left(Z_i\right|Z_{<i})$$

and

$$H\left(X\right)=\sum _{i}H\left(X_i\right|X_{<i})\le log\left(\right|\mathcal{X}\left|\right),H\left(Y\right)=\sum _{i}H\left(Y_i\right|Y_{<i})\le log\left(\right|\mathcal{Y}\left|\right).$$

Hence, it suffices to prove that

$$H\left(Z\right)\le {\displaystyle \frac{r+s}{2r}}\left(H\left(X\right)+H\left(Y\right)\right)+n(log(r+s)-s/r).$$

(6)

By noting that any fixed value of $Z_{<i}$ determines the values of $X_{<i}$ and $Y_{<i}$, we have

$$H\left(X_i\right|Z_{<i})\le H\left(X_i\right|X_{<i}),$$

and

$$H\left(Y_i\right|Z_{<i})\le H\left(Y_i\right|Y_{<i}),$$

so inequality (6) will follow from this. □

Claim 1.

Let $Past=z_{<i}$ be any specific value that $Z_{<i}$ attains. Then,

$$H\left(Z_i\right|Z_{<i}=Past)\le {\displaystyle \frac{r+s}{2r}}\left(H\left(X_i\right|Z_{<i}=Past)+H\left(Y_i\right|z_{<i}=Past)\right)+log(r+s)-{\displaystyle \frac{s}{r}}$$

Proof. 

We use the condition that $Z_{<i}=Past$, and through manipulating the notation, we drop the dependency in the notation so that, e.g., we write $H\left(X\right)$ and $H\left(Z\right|X)$ rather than $H\left(X\right|Z_{<i}=Past)$ and $H\left(Z\right|X,Z_{<i}=Past)$, etc. We also drop the index _i and write X for $X_i$, etc., so, using the new notation, we want to prove (for all integers $r\ge s\ge 0$) that

$$H\left(Z\right)\le {\displaystyle \frac{r+s}{2r}}\left(H\left(X\right)+H\left(Y\right)\right)+log(r+s)-{\displaystyle \frac{s}{r}}$$

Note that $H\left(Z\right)=H(X,Y)+H\left(Z\right|X,Y)=H(X,Y)+logr(Pr[X=Y])+logs(Pr[X\ne Y])$, so we need to prove that

$${\displaystyle \frac{r+s}{2r}}\left(H\left(X\right)+H\left(Y\right)\right)-H(X,Y)-logr\left(Pr[X=Y]\right)-logs\left(Pr[X\ne Y]\right)+log(r+s)-{\displaystyle \frac{s}{r}}\ge 0$$

Since this expression is invariant when r and s are multiplied by any positive constant, we can set $r=1$ and denote $\delta :=s/r$. Next, for a joint distribution of X and Y over ${\{0,1\}}^2$ and $(i,j)\in {\{0,1\}}^n$, let $P_{ij}=Pr[X=i,Y=j]$. Hence, we want to prove that

$$F_{\delta }\left(\begin{array}{cc}{P}_{01}& P_{11}\\ P_{00}& P_{10}\end{array}\right):={\displaystyle \frac{1+\delta }{2}}\left(H\left(X\right)+H\left(Y\right)\right)-H(X,Y)-log\delta \left(Pr[X\ne Y]\right)+log(1+\delta )-\delta \ge 0$$

(7)

We know (and can check directly from the formula) that this equality holds when $\mathcal{X}=\mathcal{Y}={\{0,1\}}^n$, in which case we have

$$\left(\begin{array}{cc}{P}_{01}& P_{11}\\ P_{00}& P_{10}\end{array}\right)=\left(\begin{array}{cc}{\displaystyle \frac{\delta }{2+2\delta }}& {\displaystyle \frac{1}{2+2\delta }}\\ {\displaystyle \frac{1}{2+2\delta }}& {\displaystyle \frac{\delta }{2+2\delta }}\end{array}\right).$$

(8)

We wish to show that this is a unique minimum. To simplify the notation (and save using indices), we denote

$$\left(\begin{array}{cc}{P}_{01}& P_{11}\\ P_{00}& P_{10}\end{array}\right):=\left(\begin{array}{cc}a& b\\ c& d\end{array}\right),$$

and attempt to minimise $F_{\delta }\left(\begin{array}{cc}a& b\\ c& d\end{array}\right)$ under the constraints

$$a,b,c,d\ge 0 \mathrm{and} a+b+c+d=1.$$

Using Lagrange multipliers, we deduce (after some simple cancelations) that at a local minimum in the interior of the region in question, one must have

$${\displaystyle \frac{\delta }{a}}{\left[(a+b)(a+c)\right]}^{(1+\delta )/2}=$$

(9)

$${\displaystyle \frac{1}{b}}{\left[(a+b)(b+d)\right]}^{(1+\delta )/2}=$$

(10)

$${\displaystyle \frac{1}{c}}{\left[(a+c)(c+d)\right]}^{(1+\delta )/2}=$$

(11)

$${\displaystyle \frac{\delta }{d}}{\left[(c+d)(b+d)\right]}^{(1+\delta )/2}.$$

(12)

From the fact that (9) · (12) = (10) · (11), we obtain

$$ad={\delta }^{2}bc$$

(13)

Next, we plug (13) and the restriction $a+b+c+d=1$ into Equation (9) = (12). This yields

$${\left({\displaystyle \frac{d}{a}}\right)}^{(1-\delta )/2}={\left[{\displaystyle \frac{1+({\delta }^{-2}-1)a}{1+({\delta }^{-2}-1)d}}\right]}^{(1+\delta )/2}.$$

(14)

Note that for every fixed value of $b+c$, the value of $a+d$ is fixed, so letting d grow from 0 to $1-b-c$ as $a=1-b-c-d$ decreases from $1-b-c$ to 0, we see that the left-hand side of (14) is increasing and the right-hand side is decreasing. Hence, there exists a single solution, which, by inspection, is $a=d$. □

We now know that $a=d$ and that $bc={\delta }^{-2}{a}^2$ and $b+c=1-2a$. This shows that b and c are the roots of the quadratic equation $X^2-(1-2a)X+{\delta }^{-2}{a}^2=0$. Plugging these roots into Equation (10) = (11) and using $a=d$ yield the following equation for a:

$${\displaystyle \frac{[1-2a+S\left(a\right)]{[1-S\left(a\right)]}^{1+\delta }}{[1-2a-S\left(a\right)]{[1+S\left(a\right)]}^{1+\delta }}}=1,$$

(15)

where $S\left(a\right)$ denotes $\sqrt{1-4a+4(1-{\delta }^{-2})a^2}$. Now, a can take on values between 0 and $1/2$ as long as $S\left(a\right)\ge 0$, so the relevant range is $0\le a\le {\displaystyle \frac{\delta }{2+2\delta }}$. When $a={\displaystyle \frac{\delta }{2+2\delta }}$, as required, $S\left(a\right)=0$ and Equation (15) clearly holds. An elementary calculation shows that for all $\delta \in [0,1]$, the left-hand side of (15) is a decreasing function of a in the interval $[0,{\displaystyle \frac{\delta }{2+2\delta }}]$, so $(a,b,c,d)$ is determined, and there is a unique internal minimum in the region, which we are exploring. (A meticulous reader may check that the derivative of the left-hand side of (15) according to a is

$${\displaystyle \frac{16a^2(1-\delta ){\left(1-S\left(a\right)\right)}^{\delta }{\left(\mathbf{1}+S\left(a\right)\right)}^{-\delta -2}(2a(\delta +1)-\delta )}{{\delta }^3{\left(S\left(a\right)+2a-1\right)}^{2}S\left(a\right)}},$$

whose sign is determined by $2a(\delta +1)-\delta$, which is negative for all $a\in (0,{\displaystyle \frac{\delta }{2+2\delta }})$).

What about points on the boundary of the region? We claim that there can be no minima with negative values on the boundary. We deal with two cases: the case of $\left(\begin{array}{cc}a& b\\ c& 0\end{array}\right)$ with $a>0$ (and the different rotations of this) and the case of $\left(\begin{array}{cc}0& b\\ c& 0\end{array}\right)$ and its rotation (note that $F_{\delta }\left(\begin{array}{cc}0& y\\ z& 0\end{array}\right)\le F_{\delta }\left(\begin{array}{cc}y& 0\\ 0& z\end{array}\right)$).

In the first case, there is a nearby point in the interior of the region, where the value of $F_{\delta }$ is smaller. This is because the derivative of

$$F_{\delta }\left(\begin{array}{cc}a-t& b\\ c& t\end{array}\right)$$

with respect to t at $t=0$ is minus infinity (as one of the summands being derived is $tlog\left(t\right)$). In the second case, the value of $F_{\delta }$ is non-negative,

$$F_{\delta }\left(\begin{array}{cc}0& b\\ 1-b& 0\end{array}\right)=\delta (blog(1/b)+(1-b)log(1/(1-b)))+log(1+\delta )-\delta ,$$

as ${log}_2(1+\delta )\ge \delta$ for all $\delta \in [0,1]$.

2.2. The General (Non-Boolean) Case

The general case is actually a minor extension of the Boolean one, which follows by adding one more coordinate to each of the random variables X and Y.

Theorem 3.

Let $ϵ\in (0,1)$, and let $f,g:{\{0,1\}}^n\to {\mathbf{R}}_{\ge 0}$. As before, let $X,Y$ be an ϵ-correlated pair of random variables with X uniformly distributed over ${\{0,1\}}^n$. Then,

$$E\left[f\left(X\right)g\left(Y\right)\right]\le {\left|f\right|}_{1+ϵ}{\left|g\right|}_{1+ϵ}.$$

(16)

Proof. 

By continuity, it suffices to consider the case where all values of f and g are rational, and by homogeneity, we can clear common denominators and assume that they are integer-valued. Now, we wish to prove a slight extension of (5), namely

$$log\left(\sum _{x\in {\{0,1\}}^n}\sum _{y\in {\{0,1\}}^n}{r}^{a(x,y)}{s}^{d(x,y)}f\left(x\right)g\left(y\right)\right)\le$$

(17)

$$n(log(r+s)-s/r)+{\displaystyle \frac{r+s}{2r}}\left(log\left(\sum _{x\in {\{0,1\}}^n}f{\left(x\right)}^{{\displaystyle \frac{2r}{r+s}}}\right)+log\left(\sum _{y\in {\{0,1\}}^n}f{\left(y\right)}^{{\displaystyle \frac{2r}{r+s}}}\right)\right).$$

(18)

To this end, we now define as before the random variables $X,Y$ and Z, add two more random integer variables a and b, and take $(X,Y,Z,A,B)$ uniformly, with $(X,Y,Z)$ defined as before and the additional constraint that $A\in \{1,\dots ,f(X\left)\right\}$ and $B\in \{1,\dots ,g(Y\left)\right\}$. Now, (17) is precisely $H(Z,A,B)=H\left(Z\right)+H\left(A\right|Z)+H\left(B\right|Z)=H\left(Z\right)+E_X[log\left(f\left(X\right)\right)]+E_Y[log\left(g\left(Y\right)\right)]$. On the other hand, note that for any function t, it holds that

$$H\left(X\right)+E_X[log\left(t\left(X\right)\right)]\le log\left(\sum _{x}t\left(x\right)\right).$$

(This is the non-negativity of the Kullbak–Leibler divergence (relative entropy) of X and $t\left(X\right)$.) In particular,

$${\displaystyle \frac{r+s}{2r}}\left(H\left(X\right)\right)+E_X[log\left(f\left(X\right)\right)]\le {\displaystyle \frac{r+s}{2r}}\left(log\left(\sum _{x\in {\{0,1\}}^n}f{\left(x\right)}^{{\displaystyle \frac{2r}{r+s}}}\right)\right)$$

and

$${\displaystyle \frac{r+s}{2r}}\left(H\left(Y\right)\right)+E_Y[log\left(g\left(Y\right)\right)]\le {\displaystyle \frac{r+s}{2r}}\left(log\left(\sum _{y\in {\{0,1\}}^n}g{\left(y\right)}^{{\displaystyle \frac{2r}{r+s}}}\right)\right)$$

To complete the proof of Theorem 3, we just add $E_X[log\left(f\left(X\right)\right)]+E_Y[log\left(g\left(Y\right)\right)]$ to both sides of the main inequality that we proved in the Boolean case, namely

$$H\left(Z\right)\le {\displaystyle \frac{r+s}{2r}}\left(H\left(X\right)+H\left(Y\right)\right)+n(log(r+s)-s/r),$$

and we are done. □