Recent Progresses in Characterising Information Inequalities

Abstract

In this paper, we present a revision on some of the recent progresses made in characterising and understanding information inequalities, which are the fundamental physical laws in communications and compression. We will begin with the introduction of a geometric framework for information inequalities, followed by the first non-Shannon inequality proved by Zhang et al. in 1998 [1]. The discovery of this non-Shannon inequality is a breakthrough in the area and has led to the subsequent discovery of many more non-Shannon inequalities. We will also review the close relations between information inequalities and other research areas such as Kolmogorov complexity, determinantal inequalities, and group-theoretic inequalities. These relations have led to non-traditional techniques in proving information inequalities and at the same time made impacts back on those related areas by the introduction of information-theoretic tools.

1. Introduction

Information inequalities are the “physical laws” that characterise the fundamental limits in communications and compression. Probably the most well-known information inequalities are the nonnegativity of entropy and mutual information, extending back to Shannon [2]. They are indispensable in proving converse coding theorems and play a critical role in information theory.

To illustrate the idea about how inequalities are invoked to prove a converse, consider the following classical scenario: Alice aims to send a source message M to Bob in a hostile environment where the transmitted message may be eavesdropped by a malicious adversary Eve. In order to ensure that Eve will learn no knowledge about the source message M, Alice will encrypt it into a transmitted message X using a private key K which is known only by Bob and herself. It is well-known that in order to have perfect secrecy, the entropy of the key K is at least as large as the entropy of the message M. Such a result can be proved by invoking a few information inequalities as follows:

$$\begin{array}{cc}\hfill H\left(M\right)& \stackrel{\left(a\right)}{=}H\left(M\right|X)\hfill \end{array}$$

(1)

$$\begin{array}{cc}& \stackrel{\left(b\right)}{=}I(M;K|X)\hfill \end{array}$$

(2)

$$\begin{array}{cc}& \stackrel{\left(c\right)}{\le }H\left(K\right|X)\hfill \end{array}$$

(3)

$$\begin{array}{cc}& \stackrel{\left(d\right)}{\le }H\left(K\right)\hfill \end{array}$$

(4)

where (a) is due to perfect secrecy (i.e., M and X are independent), (b) follows from that M can be reconstructed from the key K and the encrypted message X, (c) follows from the nonnegativity of conditional entropy $H\left(K\right|M,X)$ and (d) is due to the nonnegativity of mutual information $I(X;K)$.

Besides their role in proving converse coding theorems, information inequalities are also shown to have close relations with inequalities for Kolmogorov complexities [3], group-theoretic inequalities [4], subspace rank inequalities [5], determinantal inequalities [6] and combinatorial inequalities [7]. Therefore, any new technique in characterising information inequalities will also have direct impact on these areas.

Despite its great importance, characterising information inequalities is not an easy task. It has been open for years whether there exists other information inequalities besides the nonnegativiity of entropies and mutual information. No further information inequalities were found for fifty years, until [1] reported the first “non-Shannon” information inequality. The significance of that result lay not only in the inequality itself, but also in its construction. This particular approach for construction has been the main ingredient in every non-Shannon inequality that has been subsequently discovered. Using this approach, new inequalities can be found mechanically [8] and there are in fact infinitely many such independent inequalities even when there are only four random variables involved [9]. Despite this progress, a complete characterisation is still missing however.

In this survey paper, we will review some of the major progresses in the areas of information inequalities. The organisation of the paper is as follows. In Section 2, we will first outline a geometric framework for information inequalities, based on which we will explain how a Shannon inequality can be proved mechanically. Then we will outline the proof of a non-Shannon inequality which was first proved in [1]. A geometric perspective for the proof will also be given. Next, Matúš’ series of information inequality (and its relaxation) will be discussed.

In Section 3, we will consider several “equivalent frameworks” for information inequalities. First and the most natural one is for the scenario when random variables are continuous. We will prove that information inequalities for discrete and continuous random variables are “essentially the same”. Then we will change our focus to the one-to-one relation between information inequalities, inequalities for Kolmogorov complexity, group-theoretic inequalities and inequalities for box assignments. In Section 4, we will consider two constrained classes of information inequalities, subject to the constraint respectively that random variables are induced by vector subspaces and are Gaussian. These constrained classes of information inequalities are equivalent to subspace rank inequalities and determinantal inequalities respectively.

2. Notations

Let ${\mathcal{N}}_n=\{1,\dots ,n\}$ be a finite set and $2^{{\mathcal{N}}_n}$ be its power set. If n is understood implicitly, we will simply denote ${\mathcal{N}}_n$ by $\mathcal{N}$. We define $\mathcal{H}\left[\mathcal{N}\right]$ as the set of all real functions defined on $2^{\mathcal{N}}$. Hence, $\mathcal{H}\left[\mathcal{N}\right]$ is a $2^{\left|\mathcal{N}\right|}$-dimensional Euclidean space. Elements in $\mathcal{H}\left[\mathcal{N}\right]$ are called rank functions over $\mathcal{N}$. Let ${\mathcal{X}}_1,{\mathcal{X}}_2,\cdots ,{\mathcal{X}}_n$ be nonempty sets and $\{X_1,X_2,\cdots ,X_n\}$ be n jointly distributed discrete random variables defined on ${\mathcal{X}}_1,{\mathcal{X}}_2,\cdots ,{\mathcal{X}}_n$ respectively. For any $\alpha \subseteq \mathcal{N}$, $X_{\alpha }$ denotes the joint random variable $(X_i:i\in \alpha )$ defined over ${\mathcal{X}}_{\alpha }$ (the Cartesian product of ${\mathcal{X}}_i$ for $i\in \alpha$). As an example, $X_{\{1,2\}}$ is the random variable $(X_1,X_2)$. For simplicity, the parentheses in the subscript are usually omitted, i.e., $X_{\{1,2\}}$ is written as $X_{1,2}$ (or even simply $X_{12}$).

For a discrete random variable X, $\lambda \left(X\right)$ denotes the support of the probability distribution function of X. In other words,

$$\begin{array}{cc}\hfill \lambda \left(X\right)& \triangleq \{x\in \mathcal{X}:Pr(X=x)>0\}\hfill \end{array}$$

The (discrete) entropy of X, denoted by $H\left(X\right)$, is defined as

$$-\sum _{x\in \lambda \left(X\right)}p\left(x\right)logp\left(x\right)$$

where p is the probability distribution of X. We will also use the following conventions. Singletons and sets with one element are not distinguished. For any set $\{Y_i,i\in \mathcal{N}\}$ and subset $\alpha \subseteq \mathcal{N}$, $Y_{\alpha }$ denotes the subset $\{Y_i,i\in \alpha \}$.

3. A Framework for Information Inequalities

Let $\{X_i,i\in \mathcal{N}\}$ be a set of discrete random variables. It induces a rank function h which is defined as follows: For any $\alpha \subseteq \mathcal{N}$,

$$\begin{array}{c}\hfill h\left(\alpha \right)\triangleq H\left(X_{\alpha }\right).\end{array}$$

(5)

We call h the entropy function induced by $\left\{X_1,\dots ,X_n\right\}$. For any function h in $\mathcal{H}\left[\mathcal{N}\right]$, we define

$$\begin{array}{cc}\hfill h\left(\alpha \right|\beta )& \triangleq h(\alpha \cup \beta )-h\left(\beta \right),\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill I_h(\alpha ;\beta )& \triangleq h\left(\alpha \right)+h\left(\beta \right)-h(\alpha \cup \beta )-h(\alpha \cap \beta ).\hfill \end{array}$$

(7)

If h is the entropy function induced by random variables $\{X_i,i\in \mathcal{N}\}$, then $h\left(\alpha \right|\beta )$ is the conditional entropy $H\left(X_{\alpha }\right|X_{\beta })$ and $I_h(\alpha ;\beta )$ is the mutual information $I(X_{\alpha };X_{\beta }|X_{\alpha \cap \beta })$.

All entropy functions must satisfy the following polymatroidal axioms.

$$\begin{array}{cc}\hfill r(\varnothing )& =0\hfill \end{array}$$

(R1)

$$\begin{array}{cc}\hfill \alpha \subseteq \beta & \Rightarrow h\left(\alpha \right)\le h\left(\beta \right)\hfill \end{array}$$

(R2)

$$\begin{array}{cc}\hfill h(\alpha \cup \beta )+h(\alpha \cap \beta )& \le h\left(\alpha \right)+h\left(\beta \right).\hfill \end{array}$$

(R3)

The second axiom (R2) corresponds to that conditional entropy is nonnegative and the third axiom (R3) corresponds to that the conditional mutual information between $X_{\alpha }$ and $X_{\beta }$ given $X_{\alpha \cap \beta }$ is nonnegative.

3.1. Geometric Framework

Characterisation of entropic functions is one of the most important and challenging problems in information theory. In the following, we will review the geometric framework proposed in [10] which has greatly simplified our understanding about information inequalities.

A function $h\in \mathcal{H}\left[\mathcal{N}\right]$ is called weakly entropic if there exists $\delta >0$ such that $\delta ·h$ is entropic, and is called almost entropic if it is the limit of a sequence of weakly entropic functions. Let ${\Gamma }^{*}\left(\mathcal{N}\right)$ be the set of all entropic functions and ${\overline{\Gamma }}^{*}\left(\mathcal{N}\right)$ be its closure. Then ${\overline{\Gamma }}^{*}\left(\mathcal{N}\right)$ is a closed and convex cone, and in fact is the set of all almost entropic functions. Compared to ${\Gamma }^{*}\left(\mathcal{N}\right)$, its closure ${\overline{\Gamma }}^{*}\left(\mathcal{N}\right)$ is more manageable. In fact, for many application, it is sufficient to consider ${\overline{\Gamma }}^{*}\left(\mathcal{N}\right)$. The following proves that characterising all linear information inequalities is equivalent to characterising the set ${\overline{\Gamma }}^{*}\left(\mathcal{N}\right)$.

Theorem 1

(Yeung [10]) An information inequality ${\sum }_{\alpha \subseteq \mathcal{N}}{c}_{\alpha }H\left(X_{\alpha }\right)\ge 0$ is valid (i.e., holds for all discrete random variables) if and only if

$$\sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }h\left(\alpha \right)\ge 0,\forall h\in {\overline{\Gamma }}^{*}\left(\mathcal{N}\right).$$

Unfortunately, ${\overline{\Gamma }}^{*}\left(\mathcal{N}\right)$ is still extremely difficult to characterise explicitly for $n\ge 4$. As we shall see, the cone is not polyhedral and hence cannot be defined by a finite number of linear inequalities. Theorem 1 offers a geometric perspective in understanding information inequalities. Based on the theorem, Yan et al. [11] wrote the software called Information-Theoretic Inequality Prover (ITIP) which can mechanically verify all Shannon inequalities.

The idea behind ITIP is very simple: Suppose we have a cone ${\rm Y}$ of $\mathcal{H}\left[\mathcal{N}\right]$ such that ${\Gamma }^{*}\left(\mathcal{N}\right)\subseteq {\rm Y}$. Consider an information inequality

$$\begin{array}{c}\hfill \sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }H\left(X_{\alpha }\right)\ge 0.\end{array}$$

(8)

Suppose one can verify that

$$\sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }h\left(\alpha \right)\ge 0,\forall h\in {\rm Y}.$$

Then by Theorem 1, the information inequality (8) will be valid. In other words, if the minimum of the following optimisation problem is nonnegative,

$$\begin{array}{cc}& \text{Minimise}\sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }h\left(\alpha \right)\hfill \\ & \text{subject}\text{to}\hfill \\ & h\in {\rm Y},\hfill \end{array}$$

then the information inequality (8) is valid.

As ${\rm Y}$ is a cone (hence, $\delta h\in {\rm Y}$ for all $\delta \ge 0$ and $h\in {\rm Y}$), it is only required to test if the origin $\mathbf{0}$ is a global minimum or not in the above optimisation problem. Furthermore, as the optimisation problem is convex, the optimality of $\mathbf{0}$ can be verified by checking the Karush–Kuhn–Tucker (KKT) condition.

In ITIP, ${\rm Y}$ is chosen as the cone $\Gamma \left(\mathcal{N}\right)$ whose elements are all rank functions h that satisfies the polymatroidal axioms (R1)-(R3). By picking such a cone, the ITIP can prove all inequalities that are implied by the three axioms (or equivalently, all Shannon inequalities).

3.2. Non-Shannon Inequalities

It has been an open question for many years whether there exist information inequalities that are not implied by Shannon’s information inequalities. This question was finally answered in [1] where non-Shannon type inequalities were constructed explicitly. The proof was based on the use of auxiliary random variables. This turns out to be a very powerful technique. In fact, all subsequently discovered non-Shannon type information inequalities are essentially proved by the same technique.

Theorem 2

(Non-Shannon’s inequality [1]) Let $\{X_1,X_2,X_3,X_4\}$ be random variables. Then

$$\begin{array}{c}\hfill 2I(X_3;X_4)\le I(X_1;X_2)+I(X_1;X_3,X_4)+3I(X_3;X_4|X_1)+I(X_3;X_4|X_2).\end{array}$$

(9)

Or equivalently, if h is entropic, then

$$\begin{array}{c}\hfill 2I_h(3;4)\le I_h(1;2)+I_h(1;3,4)+3I_h(3;4|1)+I_h(3;4|2).\end{array}$$

(10)

The information inequality in Theorem 2 is a non-Shannon’s inequality because one can construct a rank function $h\in \mathcal{H}\left({\mathcal{N}}_4\right)$ such that (1) h satisfies all the polymatroidal axioms (R1)-(R3) and (2) h violates the inequality (10)

To illustrate the technique in proving new inequalities, we will sketch the proof for Theorem 2. Further details can be found in [1,12].

Sketch of proof of Theorem 2:

Let h be the entropy function induced by a set of discrete random variables $\{X_1,X_2,X_3,X_4\}$ whose underlying distribution is p. Construct two auxiliary random variables $X_1^{\prime }$ and $X_2^{\prime }$ such that

$$\begin{array}{c}\hfill Pr(x_1,x_2,x_3,x_4,x_1^{\prime },x_2^{\prime })=\left\{\begin{array}{cc}p(x_3,x_4)p(x_1,x_2|x_3,x_4)p(x_1^{\prime },x_2^{\prime }|x_3,x_4)\hfill & \text{if}p(x_3,x_4)>0\hfill \\ 0\hfill & \text{if}p(x_3,x_4)=0.\hfill \end{array}\right.\end{array}$$

(11)

It is easy to see that the marginals of $\{X_1,X_2,X_3,X_4\}$ and $\{X_1^{\prime },X_2^{\prime },X_3,X_4\}$ are the same. By invoking the basic Shannon inequalities (involving six random variables), we can prove that

$$\begin{array}{c}I(X_3;X_4)-I(X_3;X_4|X_1)-I(X_3;X_4|X_2)\\ =I(X_1;X_2^{\prime })-I(X_1;X_2^{\prime }|X_4)-I(X_1;X_2^{\prime }|X_3)-I(X_3;X_4|X_1,X_2^{\prime }).\end{array}$$

(12)

Hence,

$$\begin{array}{c}\hfill I(X_3;X_4)-I(X_3;X_4|X_1)-I(X_3;X_4|X_2)\le I(X_1;X_2^{\prime }).\end{array}$$

(13)

Similarly, we can also prove that

$$\begin{array}{c}\hfill I(X_3;X_4)-2I(X_3;X_4|X_1)\le I(X_1;X_1^{\prime }),\end{array}$$

(14)

and consequently,

$$\begin{array}{c}\hfill 2I(X_3;X_4)-3I(X_3;X_4|X_1)-I(X_3;X_4|X_2)\le I(X_1;X_1^{\prime })+I(X_1;X_2^{\prime }).\end{array}$$

(15)

Again, by invoking only Shannon’s inequalities, it can be proved that

$$\begin{array}{cc}\hfill I(X_1;X_1^{\prime })+I(X_1;X_2^{\prime })& \le I(X_1;X_3,X_4)+I(X_1^{\prime };X_2^{\prime })\hfill \end{array}$$

(16)

$$\begin{array}{cc}& =I(X_1;X_3,X_4)+I(X_1;X_2)\hfill \end{array}$$

(17)

Combining (15) and (17), the theorem is proved.Ⅰ

Remark: In the above proof of Theorem 2, the non-Shannon inequality is proved by invoking only a sequence of Shannon inequalities. This seems impossible at the first glance, as by definition, non-Shannon inequalities are all inequalities that are not implied by Shannon inequalities. The trick however is to apply Shannon inequalities over a larger set of random variables.

Using the geometric framework obtained earlier, we will describe in the following a “geometric interpretation” for the proof of the non-Shannon’s inequality.

Consider a set $\mathcal{M}$ such that $\mathcal{N}\subseteq \mathcal{M}$. Let $h\in \mathcal{H}\left[\mathcal{M}\right]$. We define ${\mathrm{proj}}_{\mathcal{N}}\left[h\right]$ as a function $g\in \mathcal{H}\left[\mathcal{N}\right]$ such that

$$g\left(\alpha \right)=h\left(\alpha \right)$$

for all $\alpha \subseteq \mathcal{N}$. Similarly, for any subset $\mathcal{A}$ of $\mathcal{H}\left[\mathcal{M}\right]$, ${\mathrm{proj}}_{\mathcal{N}}\left[\mathcal{A}\right]$ is the following subset

$${\mathrm{proj}}_{\mathcal{N}}\left[\mathcal{A}\right]\triangleq \{{\mathrm{proj}}_{\mathcal{N}}\left[h\right]:h\in \mathcal{A}\}.$$

Now, suppose that one can construct two cones ${\rm Y}$ and $\mathcal{C}$ such that

• ${\Gamma }^{*}\left(\mathcal{M}\right)\subseteq {\rm Y}$;
• For any $g\in {\Gamma }^{*}\left(\mathcal{N}\right)$, there exists a $h\in {\Gamma }^{*}\left(\mathcal{M}\right)\cap \mathcal{C}$ such that $g={\mathrm{proj}}_{\mathcal{N}}\left[h\right]$. Or equivalently, ${\Gamma }^{*}\left(\mathcal{N}\right)\subseteq {\mathrm{proj}}_{\mathcal{N}}[{\Gamma }^{*}\left(\mathcal{M}\right)\cap \mathcal{C}]$.

From the conditions 1 and 2, we have

$${\Gamma }^{*}\left(\mathcal{N}\right)\subseteq {\mathrm{proj}}_{\mathcal{N}}[{\rm Y}\cap \mathcal{C}].$$

Again, using Theorem 1, we can prove that an information inequality

$$\begin{array}{c}\hfill \sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }H\left(X_{\alpha }\right)\ge 0\end{array}$$

(18)

is valid if

$$\sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }g\left(\alpha \right)\ge 0,\forall g\in {\mathrm{proj}}_{\mathcal{N}}(\mathcal{C}\cap {\rm Y}).$$

Equivalently, the inequality (18) is valid if the minimum of the following linear program is zero.

$$\begin{array}{cc}\hfill & \text{Minimise}\sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }h\left(\alpha \right)\hfill \end{array}$$

$$\begin{array}{cc}& \text{subject}\text{to}\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill & h\in \mathcal{C}\cap {\rm Y}.\hfill \end{array}$$

Remark: Instead of verifying if an information inequality is valid or not, we can also use the Fourier-Motzkin elimination method to find all linear inequalities that defines the cone ${\mathrm{proj}}_{\mathcal{N}}(\mathcal{C}\cap {\rm Y})$. Clearly, each such inequality corresponds to a valid information inequality over $\{X_i,i\in \mathcal{N}\}$.

Now, we will revisit the non-Shannon inequality in Theorem 2. Let $\mathcal{N}=\{1,2,3,4\}$ and $\mathcal{M}=\{1,2,3,4,1^{\prime },2^{\prime }\}$. Given any random variables $\left\{X_1,\dots ,X_4\right\}$, construct two random variables $X_1^{\prime }$ and $X_2^{\prime }$ such that the probability distribution of $\{X_1,X_2,X_3,X_4,X_1^{\prime },X_2^{\prime }\}$ is given by (11). Let g be the entropy function of $\left\{X_1,\dots ,X_4\right\}$ and h be the entropy function of $\{X_1,X_2,X_3,X_4,X_1^{\prime },X_2^{\prime }\}$. Then it is easy to see that for all $i\in \{1,2\}$ and $\beta \subseteq \{3,4\}$,

$$I_h(1,2;1^{\prime },2^{\prime }|3,4)=0,h(1,2,\beta )=h(1^{\prime },2^{\prime },\beta ),h(i,\beta )=h(i^{\prime },\beta )$$

and ${\mathrm{proj}}_{\mathcal{N}}\left[h\right]=g$.

Let

$$\mathcal{C}\triangleq \left\{\begin{array}{cc}h\in \mathcal{H}\left[\mathcal{M}\right]:\hfill & I_h(1,2;1^{\prime },2^{\prime }|3,4)=0,h(1,2,\beta )=h(1^{\prime },2^{\prime },\beta ),\hfill \\ & h(i,\beta )=h(i^{\prime },\beta ),\forall i\in \{1,2\}\text{and}\beta \subseteq \{3,4\}\hfill \end{array}\right\}$$

and ${\rm Y}=\Gamma \left(\mathcal{M}\right)$ (which is the set of all functions h that satisfies the polymatroidal axioms). Then clearly ${\Gamma }^{*}\left(\mathcal{M}\right)\subseteq \Gamma \left(\mathcal{M}\right)$ and ${\Gamma }^{*}\left(\mathcal{N}\right)\subseteq {\mathrm{proj}}_{\mathcal{N}}[{\rm Y}\cap \mathcal{C}]$. It can be numerically verified that the minimum of the linear program in (19) is zero when the information inequality is the non-Shannon inequality (9). Consequently, the non-Shannon inequality is indeed proved.

3.3. Non-Polyhedral Property

In the pervious subsection, we have discussed a promising technique in proving (or even discovering) new information inequalities. Using the same technique proposed in [1], more and more linear information inequalities have been discovered [8,13,14,15]. Later in [9], Matúš obtained a countable infinite set of linear information inequalities for a set of four random variables. Using the same set of inequalities, Matúš further proved that ${\overline{\Gamma }}^{*}\left({\mathcal{N}}_4\right)$ is not a polyhedral. In the following, we will review Matúš’ inequalities and its relaxation.

Remark: The non-polyhedral property of ${\overline{\Gamma }}^{*}\left({\mathcal{N}}_4\right)$ was later used by [16] to show that the set of achievable tuples of a network is in general also non-polyhedral. As a result, this proved that the Linear Programming bounds is not tight in general.

Theorem 3 (Matúš)

Let $s\in {\mathbb{Z}}^{+}$ and $g\in {\Gamma }^{*}\left({\mathcal{N}}_4\right)$. Then

$$\begin{array}{c}\hfill s\left({\square }_{12,34}g+{▵}_{34|2}g+{▵}_{24|3}g\right)+{▵}_{23|4}g+\frac{s(s-1)}{2}\left({▵}_{24|3}g+{▵}_{34|2}g\right)\ge 0\end{array}$$

(20)

where for any distinct elements $i,j,k\in {\mathcal{N}}_4$,

$$\begin{array}{cc}\hfill {▵}_{ij|k}g& \triangleq I_g(X_i;X_j|X_k),\hfill \\ \hfill {\square }_{12,34}g& \triangleq g\left(13\right)+g\left(23\right)+g\left(14\right)+g\left(24\right)+g\left(34\right)\hfill \\ & -g\left(12\right)-g\left(3\right)-g\left(4\right)-g\left(134\right)-g\left(234\right).\hfill \end{array}$$

While Matúš proved a series of linear information inequalities, it is sometimes difficult to use these infinitely number of inequalities at the same time. In [17], the series of Matúš’ inequalities is relaxed to a single non-linear inequality.

Remark: Using one single nonlinear inequality, it can be proved that the set of all almost entropic functions is not polyhedral.

Theorem 4

(Quadratic information inequality [17]) Let $g\in {\overline{\Gamma }}^{*}\left(\mathcal{N}\right)$,

$$\begin{array}{cc}\hfill a\left(g\right)& \triangleq \frac{1}{2}\left({▵}_{24|3}g+{▵}_{34|2}g\right)\hfill \\ \hfill b\left(g\right)& \triangleq {\square }_{12,34}g+{▵}_{34|2}g+{▵}_{24|3}g\hfill \\ \hfill c\left(g\right)& \triangleq {▵}_{23|4}g\hfill \\ \hfill w\left(g\right)& \triangleq \left\{\begin{array}{cc}-\frac{b\left(g\right)-a\left(g\right)}{2a\left(g\right)}\hfill & ifa\left(g\right)>0\hfill \\ 0\hfill & \mathit{otherwise}.\hfill \end{array}\right.\hfill \end{array}$$

(21)

If $b\left(g\right)\le 2a\left(g\right)$, then

$$\begin{array}{c}\hfill {(b\left(g\right)-a\left(g\right))}^2-4a\left(g\right)c\left(g\right)\le min(4a{\left(g\right)}^2{(w\left(g\right)-\lfloor w\left(g\right)\rfloor )}^2,4a{\left(g\right)}^2{(\lceil w\left(g\right)\rceil -w\left(g\right))}^2)\end{array}$$

(22)

and consequently,

$$\begin{array}{c}\hfill {\left({\square }_{12,34}g+\frac{{▵}_{24|3}g+{▵}_{34|2}g}{2}\right)}^2-2({▵}_{24|3}g+{▵}_{34|2}g){▵}_{32|4}g\le \frac{{({▵}_{24|3}g+{▵}_{34|2}g)}^2}{4}.\end{array}$$

(23)

Remark: Subject to the constraint that $b\left(g\right)>2a\left(g\right)$, then the series of linear inequalities (20) is implied by the Shannon inequalities. Therefore, the constraint (i.e., $b\left(g\right)>2a\left(g\right)$) we imposed on Theorem 4 is not critical.

Conjecture 1

(20) holds for all $s\ge 0$. Consequently, if $b\left(g\right)\le 2a\left(g\right)$, then

$${(b\left(g\right)-a\left(g\right))}^2-4a\left(g\right)c\left(g\right)\le 0.$$

4. Equivalent Frameworks

In the previous section, we have described a framework for information inequalities for discrete random variables. We have also demonstrated the common proving technique. In this section, we will construct several different frameworks which are “equivalent” or “almost equivalent” to the earlier one. These equivalence relations among different frameworks will turn out to be very useful in deriving new information theoretic tools.

4.1. Differential Entropy

The previous framework for information inequalities assumes that all random variables are discrete. A very natural extension of the framework is thus to relax the restriction by allowing random variables to be continuous. To achieve this goal, we will first need an analogous definition of discrete entropy in the domain of continuous random variables.

Definition 1 (Differential entropies)

Let $\{X_i,i\in \mathcal{N}\}$ be a set of continuous random variables such that $X_i$ are real numbers. For any $\alpha \subseteq \mathcal{N}$, let $f_{\alpha }(x_i,i\in \alpha )$ be the density functions for $(X_i,i\in \alpha )$. Then the differential entropy of $(X_i,i\in \alpha )$ is denoted by

$$H\left(X_{\alpha }\right)\triangleq -\int f\left(x_{\alpha }\right)logf\left(x_{\alpha }\right)d_{x_{\alpha }}.$$

Remark: For notation simplicity, we abuse our notations by using $H\left(X\right)$ to denote both discrete and differential entropies. However, its exact meaning should be clear from the context.

Discrete and differential entropies shared similar and dissimilar properties. The main difference is that differential entropy can be negative, unlike discrete entropy. However, mutual information and its conditional counterpart (by defined analogously as in (7)) remain nonnegative. In fact, as we shall see, the sets of information inequalities for discrete and continuous random variables are almost the same.

Definition 2 (Balanced inequalities)

An information inequality ${\sum }_{\alpha \subseteq \mathcal{N}}{c}_{\alpha }H\left(X_{\alpha }\right)\ge 0$ (for either discrete or continuous random variables) is called balanced if for all $n\in \mathcal{N}$, ${\sum }_{\alpha \subseteq \mathcal{N}:n\in \alpha }{c}_{\alpha }=0$.

For any information inequality ${\sum }_{\alpha \subseteq \mathcal{N}}{c}_{\alpha }H\left(X_{\alpha }\right)\ge 0$ or expression ${\sum }_{\alpha \subseteq \mathcal{N}}{c}_{\alpha }H\left(X_{\alpha }\right)$, its $n^{th}$ residual weight $r_n$ is defined as

$$\begin{array}{c}\hfill r_n\triangleq \sum _{\alpha \subseteq \mathcal{N}:n\in \alpha }{c}_{\alpha }.\end{array}$$

(24)

Clearly, an information inequality is balanced if and only if $r_n=0$ for all $n\in \mathcal{N}$.

Example 1

The residual weights $r_1,r_2$ of the information inequality $H\left(X_1\right)+H\left(X_2\right)\ge 0$ are both equal to one. Hence, the inequality is not balanced.

For any information inequality ${\sum }_{\alpha \subseteq \mathcal{N}}{c}_{\alpha }H\left(X_{\alpha }\right)\ge 0$, its balanced counterpart is the following inequality

$$\begin{array}{c}\hfill \sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }H\left(X_{\alpha }\right)-\sum _{n\in \mathcal{N}}{r}_{n}H\left(X_n\right|X_i,i\ne n)\ge 0,\end{array}$$

(25)

which is balanced (as its name suggests).

Proposition 1

(Necessary and sufficiency of balanced inequalities [6]) For any valid information inequality ${\sum }_{\alpha \subseteq \mathcal{N}}{c}_{\alpha }H\left(X_{\alpha }\right)\ge 0$ , it is a valid discrete information inequality if and only if

1. 

its residual weights $r_n\ge 0$ for all n, and

2. 

its balanced counterpart is also valid.

$$\sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }H\left(X_{\alpha }\right)-\sum _{n\in \mathcal{N}}{r}_{n}H\left(X_n\right|X_i,i\ne n)\ge 0.$$

Consequently, all valid discrete information inequalities are implied by the set of all valid balanced inequalities and the nonnegativity of (conditional) entropies.

It turns out that this set of balanced information inequalities also play the same significant role for inequalities involving continuous random variables.

Theorem 5

(Equivalence [6]) All information inequalities for continuous random variables are balanced. Furthermore, a balanced information inequality

$$\sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }H\left(X_{\alpha }\right)-\sum _{n\in \mathcal{N}}{r}_{n}H\left(X_n\right|X_i,i\ne n)\ge 0$$

is valid for continuous random variable if and only if it is also valid for discrete random variables.

By Theorem 5, to characterise information inequalities, it is sufficient to consider only balanced information inequalities which are the same for either discrete or continuous random variables.

4.2. Inequalities for Kolmogorov Complexity

The second framework we will describe is quite different from the earlier information-theoretic frameworks. For information inequalities, the objects of interest are random variables. However, for the following Kolmogorov complexity framework, the objects of interest are deterministic strings instead.

To understand what Kolmogorov complexity is, let us consider the following example: Suppose that $x_1$ and $x_2$ are the following binary strings

$$\begin{array}{cc}\hfill x_1& =1111111111111111111111111111111111111111111111111111111111111111\hfill \end{array}$$

(26)

$$\begin{array}{cc}\hfill x_2& =0110010101010100010001010100101101001011101010101001011011101011.\hfill \end{array}$$

(27)

Kolmogorov complexity of a string x (denoted by $K⟨x⟩$) is the minimal program length required to output that string [18] In the above example, it is clear that the Kolmogorov complexity of $x_1$ is much smaller than that of $x_2$ (which is obtained by flipping a fair coin).

Although the objects of interest are different, [3] proved a surprising result that inequalities for Kolmogorov complexities and for entropies are essentially the same.

Theorem 6

(Equivalence [3]) An information inequality (for discrete random variable) ${\sum }_{\alpha }{c}_{\alpha }H(X_i,i\in \alpha )\ge 0$ is valid if and only if the corresponding Kolmogorov complexity inequality defined below

$$\sum _{\alpha }{c}_{\alpha }K(x_i,i\in \alpha )\ge 0$$

is also valid.

4.3. A Group-Theoretic Framework

Besides Kolmogorov complexities, information inequalities are also closely related to group-theoretic inequalities [4]. To understand their relation, we first illustrate how to construct a random variable from a subgroup.

Definition 3 (Group-theoretic construction of random variables)

Let G be a finite group and U be a random variable that takes value in G uniformly. In other words,

$$\begin{array}{c}\hfill Pr(U=i)=\frac{1}{\left|G\right|}\end{array}$$

(28)

for all $i\in G$.

For any subgroup K of G, it partitions G into $\left|G\right|/\left|K\right|$’s left (or right) coset of K in G such that each coset has exactly $\left|K\right|$’s elements. Note that, each coset can be written as the following subset for some elements $b\in G$

$$\{b\circ k:k\in K\}$$

where ∘ is the binary group operator. Let ${\Omega }_K$ be the collection of all left cosets of K in G. The subgroup K induces a random variable $X_K$, which is defined as the random left coset of K in G that contains U. In fact, $X_K$ is equal to the following coset

$$\begin{array}{c}\hfill \{U\circ k:k\in K\}.\end{array}$$

(29)

Since U is uniformly distributed over G, we can easily prove that $X_K$ is uniformly distributed over ${\Omega }_K$ and that

$$H\left(X_K\right)=log\left|G\right|/\left|K\right|.$$

The above construction of a random variable from a subgroup can be extended naturally to multiple subgroups.

Theorem 7

(Group characterisable random variables [4]) Let G be a finite group and $\{G_i,i\in \mathcal{N}\}$ be a set of subgroups of G. For each $i\in \mathcal{N}$, let $X_i$ be the random variable induced by the subgroup $G_i$ as defined above. Then for any $\alpha \subseteq \mathcal{N}$,

1. 

$H(X_i,i\in \alpha )=log\left|G\right|/|{\cap }_{i\in \alpha }{G}_i|$,

2. 

$|\lambda (X_i,i\in \alpha )|=\left|G\right|/|{\cap }_{i\in \alpha }{G}_i|$,

3. 

$(X_i,i\in \alpha )$ is uniformly distributed over its support. In other word, the value of the probability distribution function of $(X_i,i\in \alpha )$ is either zero or is a constant.

Definition 4

A function $h\in \mathcal{H}\left[\mathcal{N}\right]$ is called group characterisable if it is the entropy function of a set of random variables $\left\{X_1,\dots ,X_n\right\}$ induced by a finite group G and its subgroups $\left\{G_1,\dots ,G_n\right\}$. Furthermore, h is

1. 

representable if $\{G,G_1,\dots ,G_n\}$ are all vector space, and

2. 

abelian if G is abelian.

Clearly, random variables induced by a set of subgroups must satisfy all valid information inequalities Therefore, we have the following theorem.

Theorem 8

(Group-theoretic inequalities [4]) Let

$$\begin{array}{c}\hfill \sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }H\left(X_{\alpha }\right)\ge 0\end{array}$$

(30)

be a valid information inequality. Then for any finite group G and its subgroups $\{G_i,i\in \mathcal{N}\}$, we have

$$\begin{array}{c}\hfill \sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }log\frac{\left|G\right|}{|{\cap }_{i\in \alpha }{G}_i|}\ge 0,\end{array}$$

(31)

or equivalently,

$$\begin{array}{c}\hfill {\left|G\right|}^{{\sum }_{\alpha \subseteq \mathcal{N}}{c}_{\alpha }}\ge \prod _{\alpha \subseteq \mathcal{N}}{\left|{\cap }_{i\in \alpha }{G}_i\right|}^{c_{\alpha }}.\end{array}$$

(32)

Theorem 8 proved that we can directly “translate” any information inequality into a group-theoretic inequality. A very surprising result proved in [4] was that the the converse also holds.

Theorem 9

(Converse [4]) The information inequality (30) is valid if it is satisfied by all random variables induced by groups, or equivalently, the group-theoretic inequality (32) is valid.

Theorems 8 and 9 suggested that to prove an information inequality, it is necessary and sufficient to verify if the inequality is satisfied by all random variables induced by groups. Later, we will further illustrate how to use the two theorems to derive a group-theoretic proof for information inequalities.

In the following, we will further prove that many statistical properties of random variables induced by groups will have analogous algebraic interpretations.

Lemma 1 (Properties of group induced random variables)

Suppose that $\{X_i,i\in \mathcal{N}\}$ is a set of random variables induced by a finite group G and its subgroups $\{G_i,i\in \mathcal{N}\}$. Then

1. 

(Functional dependency) $H\left(X_l\right|X_i,i\in \alpha )=0$ (i.e., $X_l$ is a function of $X_{\alpha }$) if and only if ${\cap }_{i\in \alpha }{G}_i\subseteq G_l$. Hence, functional dependency is equivalent to subset relation;

2. 

(Independency) $I(X_i;X_j|X_l)=0$ if and only if

$$\begin{array}{c}\hfill |G_i\cap G_l\left|\right|G_j\cap G_l|=|G_l\left|\right|G_i\cap G_j\cap G_l|;\end{array}$$

(33)

3. 

(Conditioning preserves group characterisation) for any fixed any $\alpha \subseteq \mathcal{N}$, the group $K\triangleq {\cap }_{i\in \alpha }{K}_i$ and its subgroups $K_i\triangleq K\cap G_i$ for $i\in \mathcal{N}$ induce a set of random variables $\{Y_i,i\in \mathcal{N}\}$ such that

$$H(Y_i,i\in \beta )=H(X_i,i\in \beta |X_j,j\in \alpha )$$

for all $\beta \subseteq \mathcal{N}$. In other words, for any group characterisable $h\in \mathcal{H}\left[\mathcal{N}\right]$, let $g\in \mathcal{H}\left[\mathcal{N}\right]$ such that

$$g\left(\beta \right)=h\left(\beta \right|\alpha )$$

for all $\beta \subseteq \mathcal{N}$. Then g is also group characterisable.

Proposition 2

(Duality [19]) Let $\left\{V_1,\dots ,V_n\right\}$ be a set of vector subspaces of $V\triangleq {\mathbb{F}}^m$ over the finite field $\mathbb{F}$. Define the following subspace $W_i$ for $i\in \mathcal{N}$:

$$\begin{array}{c}\hfill W_i=\{w\in V:v^{\top }w=0\}.\end{array}$$

(34)

Then, for any $\alpha \subseteq \mathcal{N}$,

$$dim⟨V_i,i\in \alpha ⟩=dimV-dim\bigcap _{i\in \alpha }{W}_i=log\frac{\left|V\right|}{|{\bigcap }_{i\in \alpha }{W}_i|}.$$

Hence, if $h\in \mathcal{H}\left[\mathcal{N}\right]$ such that $h\left(\alpha \right)\triangleq dim⟨V_i,i\in \alpha ⟩$ for all $\alpha \subseteq \mathcal{N}$, then h is weakly representable.

Remark: While $W^{\perp }$ and W are both subspaces of V and $dimW+dim{W}^{\perp }=dimV$, $⟨W,W^{\perp }⟩\ne V$ in general. If $\mathbb{F}=\mathbb{R}$, then $W_i$ (defined as in (34)) is the orthogonal complement of $V_i$.

Theorems 8 and 9 suggested that proving an information inequality (30) is equivalent to proving a group-theoretic inequality (32). In the following, we will illustrate the idea by providing a group-theoretic proof for nonnegativity of mutual information

$$\begin{array}{c}\hfill H\left(X_1\right)+H\left(X_2\right)\ge H(X_1,X_2).\end{array}$$

(35)

Example 2 (Group-theoretic Proof)

Let G be a finite group and $G_1$ and $G_2$ be its subgroups. Let

$$S=\{a\circ b:a\in G_1,b\in G_2\}$$

where ∘ is the binary group operator. As S is a subset of $\left|G\right|$, $\left|S\right|\le \left|G\right|$. With a simple counting argument (by removing duplications), it can be proved easily that

$$\left|S\right|=\frac{|G_1\left|\right|G_2|}{|G_1\cap G_2|}.$$

Therefore,

$$\left|G\right||G_1\cap G_2|\ge |G_1\left|\right|G_2|.$$

Finally, according to Theorems 8 and 9, the inequality (35) follows.

It is worth mentioning that Theorems 8 and 9 also suggested an information-theoretic proof for group-theoretic inequalities. For example, the following information inequality

$$\begin{array}{c}\hfill {\displaystyle H\left(X_1\right)+H\left(X_2\right)+2H(X_1,X_2)+4H\left(X_3\right)+4H\left(X_4\right)}\end{array}$$

(36)

$$\begin{array}{c}\hfill {\displaystyle +5H(X_1,X_3,X_4)+5H(X_2,X_3,X_4)}\end{array}$$

(37)

$$\begin{array}{ccc}& \le & 6H(X_3,X_4)+4H(X_1,X_3)+4H(X_1,X_4)\hfill \end{array}$$

(38)

$$\begin{array}{ccc}& & +4H(X_2,X_3)+4H(X_2,X_4),\hfill \end{array}$$

(39)

implies the following group-theoretic inequality

$$\begin{array}{c}\hfill |G_{34}{|}^6|G_{13}{|}^4|G_{14}{|}^4|G_{23}{|}^4|G_{24}{|}^4\le |G_1\left|\right|G_2\left|\right|G_3{|}^4|G_4{|}^4|G_{12}{|}^2|G_{134}{|}^5{\left|G_{234}\right|}^5\end{array}$$

(40)

The meaning of this inequality and its implications in group theory are yet to be understood.

4.4. Combinatorial Perspective

Random variables that are induced by groups have many interesting properties. One interesting property is that they are quasi-uniform in nature.

Definition 5 (Quasi-uniform random variables)

A set of random variables $\left\{X_1,\dots ,X_n\right\}$ is called quasi-uniform if for all $\alpha \subseteq \mathcal{N}$, $X_{\alpha }\triangleq (X_i,i\in \alpha )$ is uniformly distributed over its support $\lambda \left(X_{\alpha }\right)$. In other words,

$$\begin{array}{c}\hfill Pr(X_{\alpha }=x_{\alpha })=\left\{\begin{array}{cc}1/\left|\lambda \right(X_{\alpha }\left)\right|\hfill & if{x}_{\alpha }\in \lambda \left(X_{\alpha }\right)\hfill \\ 0\hfill & otherwise.\hfill \end{array}\right.\end{array}$$

(41)

Since $X_{\alpha }$ is uniformly distributed for all $\alpha \subseteq \mathcal{N}$, the entropy $H\left(X_{\alpha }\right)$ is thus equal to $log\left|\lambda \right(X_{\alpha }\left)\right|$.

According to the Asymptotic Equipartition Property (AEP) [12], for a sufficiently long sequence of independent and identically distributed random variables, the set of typical sequences has a total probability close to one and the probability of each typical sequence is approximately the same. In certain sense, quasi-uniform random variables possess the non-aymptotic equipartition property that the probabilities are completely concentrated and uniformly distributed over their supports. As a result, quasi-uniform random variables can be fully characterised by their supports (because the probability distributions are uniform over the supports). This offers a combinatorial interpretation for quasi-uniform random variables. And it turns out that this interpretation offers a combinatorial approach to proving information inequalities.

Definition 6 (Box assignment)

Let $\left\{{\mathcal{X}}_1,\dots ,{\mathcal{X}}_n\right\}$ be nonempty finite sets and $\mathcal{X}$ be their Cartesian product ${\prod }_{i=1}^n{\mathcal{X}}_i$. A box assignment $\mathcal{A}$ in $\mathcal{X}$ is a nonempty subset of ${\mathcal{X}}_{\mathcal{N}}$.

For any $\alpha \subseteq \mathcal{N}$ and $a_{\alpha }\triangleq (a_i,i\in \alpha )\in {\prod }_{i\in \alpha }{\mathcal{X}}_i$, we define

$$\begin{array}{cc}\hfill {\mathcal{A}}_{\mathcal{N}|\alpha }\left(a_{\alpha }\right)& \triangleq \left\{(x_j,j\in \mathcal{N})\in \mathcal{A}:x_i=a_i,i\in \alpha \right\},\hfill \end{array}$$

(42)

$$\begin{array}{cc}\hfill {\mathcal{A}}_{\alpha }& \triangleq \left\{(a_i,i\in \alpha ):\left|{\mathcal{A}}_{\mathcal{N}|\alpha }\left(a_{\alpha }\right)\right|\ge 1\right\}.\hfill \end{array}$$

(43)

Roughly speaking, ${\mathcal{A}}_{\mathcal{N}|\alpha }\left(a_{\alpha }\right)$ is the set of elements in $\mathcal{A}$ such that its “$i^{th}$-coordinate” is $a_i$ for $i\in \alpha$. The set ${\mathcal{A}}_{\mathcal{N}|\alpha }\left(a_{\alpha }\right)$ will be called the $a_{\alpha }$-layer of $\mathcal{A}$. And hence, ${\mathcal{A}}_{\alpha }$ contains all $a_{\alpha }$ such that the $a_{\alpha }$-layer of $\mathcal{A}$ is nonempty. And we will call ${\mathcal{A}}_{\alpha }$ the α-projection of $\mathcal{A}$.

Definition 7 (Quasi-uniform box assignment)

A box assignment $\mathcal{A}$ is called quasi-uniform if for any $\alpha \subseteq \mathcal{N}$, the cardinality of ${\mathcal{A}}_{\mathcal{N}|\alpha }\left(a_{\alpha }\right)$ is constant for all $a_{\alpha }\in {\mathcal{A}}_{\alpha }$. And we will denote the constant by $|{\mathcal{A}}_{\mathcal{N}|\alpha }|$ for simplicity.

The following proposition proves that quasi-uniform box assignment and quasi-uniform random variables are in fact equivalent.

Proposition 3

(Equivalence [7]) Let $\left\{X_1,\dots ,X_n\right\}$ be a set of quasi-uniform random variables and $\mathcal{A}$ be its probability distribution’s support. Then $\mathcal{A}$ is a quasi-uniform box assignment in ${\prod }_{i\in \mathcal{N}}{\mathcal{X}}_i$. Furthermore, for all $\alpha \subseteq \mathcal{N}$,

$$\begin{array}{c}\hfill H\left(X_{\alpha }\right)=log\left|{\mathcal{A}}_{\alpha }\right|.\end{array}$$

(44)

Conversely, for any quasi-uniform box assignment $\mathcal{A}$, there exists a set of quasi-uniform random variables $\left\{X_1,\dots ,X_n\right\}$ whose probability distribution’s support is indeed $\mathcal{A}$.

As random variables induced by groups are quasi-uniform, by Theorems 8 and 9, we have the following combinatorial interpretation for information inequalities.

Theorem 10

(Combinatorial interpretation [7]) An information inequality

$$\begin{array}{c}\hfill \sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }H\left(X_{\alpha }\right)\ge 0\end{array}$$

(45)

is valid if and only if the following box assignment inequality is valid

$$\begin{array}{c}\hfill \sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }log\left|{\mathcal{A}}_{\alpha }\right|\ge 0,\end{array}$$

(46)

or equivalently,

$$\begin{array}{c}\hfill \prod _{\alpha \subseteq \mathcal{N}}{\left|{\mathcal{A}}_{\alpha }\right|}^{c_{\alpha }}\ge 1\end{array}$$

(47)

for all quasi-uniform box assignments $\mathcal{A}$.

Again, in the following example, we will illustrate how to use the combinatorial interpretation to derive a “combinatorial proof” for information inequality.

Example 3 (Combinatorial proof)

Let $\mathcal{A}$ be a quasi-uniform box assignment in $\mathcal{X}={\mathcal{X}}_1\times {\mathcal{X}}_2$. Suppose $(a_1,a_2)\in \mathcal{A}$. Then it is obvious that $a_1\in {\mathcal{A}}_1$ and $a_2\in {\mathcal{A}}_2$. In other words, $\mathcal{A}\subseteq {\mathcal{A}}_1\times {\mathcal{A}}_2$ and consequently,

$$|{\mathcal{A}}_1|\times |{\mathcal{A}}_2|\ge |{\mathcal{A}}_{1,2}|.$$

By Theorem 10, we prove that $H\left(X_1\right)+H\left(X_2\right)\ge H(X_1,X_2)$.

4.5. Coding Perspective

We can also view a box assignment $\mathcal{A}$ as an error correcting code such that $\mathcal{A}$ is the set of all codewords. For each codeword $\left(a_1,\dots ,a_n\right)$, $a_i$ is the $i^{th}$ symbol to be transmitted across a channel. Taking this coding perspective, in the following, a box assignment will simply be called a code. Also, a code $\mathcal{C}$ is called a quasi-uniform code if $\mathcal{C}$ is a quasi-uniform box assignment. Again, each quasi-uniform code $\mathcal{C}$ will induce a set of quasi-uniform random variables $\left\{X_1,\dots ,X_n\right\}$.

For any code $\mathcal{C}$ (which is just a box assignment) and two codewords $c,c^{\prime }\in \mathcal{C}$, the Hamming distance between codewords $c\triangleq \left(c_1,\dots ,c_n\right)$ and $c^{\prime }\triangleq \left(c_1^{\prime },\dots ,c_n^{\prime }\right)$ is defined as

$$D(c,c^{\prime })\triangleq \left|\{i\in \mathcal{N}:c_i\ne c_i^{\prime }\}\right|.$$

In addition, the minimum Hamming distance of the code $\mathcal{C}$ is defined as

$$D\left(\mathcal{C}\right)\triangleq \underset{c\ne c^{\prime },c,c^{\prime }\in \mathcal{C}}{min}D(c,c^{\prime })$$

The minimum Hamming distance of a code characterises how strong the error correcting capability of the code is. Specifically, a code $\mathcal{C}$ with a minimum Hamming distance d can correct up to $\lfloor \frac{d-1}{2}\rfloor$’s symbol errors.

Example 4

Let $\mathcal{C}$ be a length-3 code containing only two codewords $(0,0,0)$ and $(1,1,1)$. The minimum Hamming distance of this code is 3 and hence can correct any single symbol error. For instance, suppose the codeword $(0,0,0)$ is transmitted. If a symbol error occurs, the receiver will receive either $(1,0,0)$, $(0,1,0)$ or $(0,0,1)$. In any case, the receiver can always determine which symbol is erroneous (by using a bounded-distance decoder) and hence can correct it.

In addition to the minimum Hamming distance, in many cases, a code’s distance profile is also of great importance: Let $\mathcal{C}$ be a code and c be a codeword in $\mathcal{C}$. The distance profile of $\mathcal{C}$ centered at c is a set of integers $A(\mathcal{C},c)\triangleq \{A_r\left(c\right):r=1,\dots ,n\}$ where

$$A_r\left(c\right)\triangleq |\{c^{\prime }\in C:D(c,c^{\prime })=r\}\left|\right\}.$$

In other words, $A_r\left(c\right)$ is the number of codewords in $\mathcal{C}$ such that their Hamming distances to the centering codeword c is r.

The profile $A(\mathcal{C},c)$ contains information about how likely a decoding error (i.e., the receiver decodes a wrong codeword) occurs if the transmitted codeword is c. In general, the distance profile $A(\mathcal{C},c)$ depends on the choice of c. A code is called distance-invariant if its distance profile $A(\mathcal{C},c)$ is independent of c. Roughly speaking, a distance-invariant code is one where the probability of decoding error is the same for all transmitted codewords $c\in \mathcal{C}$.

Theorem 11

(Distance invariance [20]) Quasi-uniform codes are distance-invariant.

Example 5 (Linear codes)

Let P be a $n-k\times n$ parity check matrix (over a finite field $\mathbb{F}$) and the code $\mathcal{C}$ is defined by

$$\mathcal{C}=\{c\in {\mathbb{F}}^n:Pc=\mathbf{0}\}.$$

Then $\mathcal{C}$ is called a linear code. Note that, for a linear code, if $c_1,c_2\in \mathcal{C}$, then $c_1+c_2$ is also contained in $\mathcal{C}$. Linear codes are quasi-uniform codes and hence are also distance invariant.

In the following, we will consider only quasi-uniform codes. For simplicity, we will assume without loss of generality that there is a zero-codeword $\mathbf{0}\in \mathcal{C}$ (by renaming). Also, for any $c\in \mathcal{C}$, we define the Hamming weight of the codeword c (denoted by $D\left(c\right)$) as $D(c,\mathbf{0})$.

Definition 8 (Weight enumerator)

The weight enumerator of a quasi-uniform code C with length n is

$$W_C(x,y)=\sum _{r=1}^n{A}_r{x}^{n-r}{y}^r$$

where x and y are indeterminates, and $A_r\triangleq A_r\left(\mathbf{0}\right)$. Using simple counting, it is easy to prove that

$$\begin{array}{c}\hfill W_C(x,y)\triangleq \sum _{c\in C}{x}^{n-D\left(c\right)}{y}^{D\left(c\right)}.\end{array}$$

(48)

In many cases, it is more convenient to work with weight enumerator than distance profile. However, conceptually, they are equivalent (i.e., they can be uniquely obtained from each other). Clearly, the weight enumerator is uniquely determined from the code $\mathcal{C}$. However, what “structural property” of the code $\mathcal{C}$ determines the weight enumerator? For example, suppose that we construct a new code from $\mathcal{C}$ by exchanging the first and the second codeword symbols. It is obvious that this modification will not affect the weight enumerator. In other words, ordering of the codeword symbols has no effects on the weight enumerator. The question therefore is: What property of a code has direct effects on the weight enumerator?

To answer the question, let us use the old perspective that a quasi-uniform code is merely a quasi-uniform box assignment (and also its associated set of quasi-uniform random variables). These random variables $\left\{X_1,\dots ,X_n\right\}$ have a simple interpretation here: Suppose a codeword $C=\left(C_1,\dots ,C_n\right)$ is randomly and uniformly selected from $\mathcal{C}$. Then $X_i$ is the $i^{th}$ symbol in the random codeword C, i.e., $X_i=C_i$. Our answer to the above question is given in the following theorem.

Theorem 12

(Generalised Greene’s Theorem [20]) Let C be a quasi-uniform code and $\left\{X_1,\dots ,X_n\right\}$ be its induced quasi-uniform random variables. Suppose that ρ is the entropy function of $\left\{X_1,\dots ,X_n\right\}$. In other words, $\rho \left(\alpha \right)=H(X_i,i\in \alpha )$. Then

$$\begin{array}{c}\hfill W_C(x,y)=\sum _{\alpha \subseteq \mathcal{N}}{2}^{\rho \left(\mathcal{N}\right)-\rho \left(\alpha \right)}{(x-y)}^{\left|\alpha \right|}{y}^{n-\left|\alpha \right|}.\end{array}$$

(49)

Remark: The Greene’s Theorem is a special case of Theorem 12 when the code $\mathcal{C}$ is a linear code.

By Theorem 12, the weight enumerator (and also the error-correcting capability) of a quasi-uniform code depends only on the entropy function induced by the codeword symbol random variables. By exploiting the relation between the entropy function of a set of quasi-uniform random variables and the weight enumerator of the induced code, we open a new door on how to harness coding theory results to derive new information theory results.

Example 6

(Code-theoretic proof) Consider a set of quasi-uniform random variables $\{X_1,X_2\}$ which induces a length-2 quasi-uniform code C. The length of the code is 2. By the Generalised Greene’s Theorem, the number of codewords which have Hamming weights 1 is given by

$$\begin{array}{c}\hfill A_1=(2^{H(X_1,X_2)-H\left(X_1\right)}+2^{H(X_1,X_2)-H\left(X_2\right)}-2).\end{array}$$

(50)

As $A_1$ is nonnegative, (50) implies that

$$\begin{array}{c}\hfill min(H\left(X_1\right),H\left(X_2\right))\le H(X_1,X_2).\end{array}$$

(51)

Finally, by Theorem 10 (a variation of which to be precise), an information inequality holds if and only if it also holds for all quasi-uniform random variables. Consequently, we prove that (51) holds for all random variables.

5. Constrained Information Inequalities

In pervious sections, we considered general information inequalities where we do not impose any constraint on the choice of random variables. In the following, we will focus on two constrained classes of information inequalities: subspace rank inequalities and determinantal inequalities.

5.1. Rank Inequalities

Let $\left\{V_1,\dots ,V_n\right\}$ be a set of vector subspaces over a field $\mathbb{F}$. A subspace rank inequality is an inequality about the rank or dimension of subspaces in the following form:

$$\begin{array}{c}\hfill \sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }dim⟨V_i,i\in \alpha ⟩\ge 0.\end{array}$$

(52)

For example, it is straightforward to prove that

$$\begin{array}{c}\hfill dim⟨V_1⟩+dim⟨V_2⟩\ge dim⟨V_1,V_2⟩,\end{array}$$

(53)

which is a direct consequence of the following identity

$$\begin{array}{c}\hfill dim⟨V_1⟩+dim⟨V_2⟩=dim⟨V_1,V_2⟩+dim{V}_1\cap V_2.\end{array}$$

(54)

Subspace rank inequalities are in fact constrained information inequalities subject to the criteria that random variables are induced by vector subspaces over a field. Clearly, all valid information inequalities (including all Shannon inequalities) are subspace rank inequalities. For example, the subspace rank inequality (53) is indeed equivalent to the nonnegativity of mutual information. Besides all these known unconstrained information inequalities, one of the most well-known subspace rank inequalities is the Ingleton inequalities [21]. A recent work [22] proved that Ingleton inequalities also include Shannon inequalities as special cases and determined the unique minimal set of Ingleton inequalities that imply all the others.

Theorem 13 (Ingleton inequality)

Suppose r is a representable polymatroid over $\mathcal{X}$. Then for every choice of subsets $X_1,{\mathcal{X}}_2,{\mathcal{X}}_3,{\mathcal{X}}_4\subseteq \mathcal{X}$

$$\begin{array}{c}0\le r({\mathcal{X}}_1\cup {\mathcal{X}}_2)+r({\mathcal{X}}_1\cup {\mathcal{X}}_3)+r({\mathcal{X}}_1\cup {\mathcal{X}}_4)+r({\mathcal{X}}_2\cup {\mathcal{X}}_3)+r({\mathcal{X}}_2\cup {\mathcal{X}}_4)\\ -r\left({\mathcal{X}}_1\right)-r\left({\mathcal{X}}_2\right)-r({\mathcal{X}}_3\cup {\mathcal{X}}_4)-r({\mathcal{X}}_1\cup {\mathcal{X}}_2\cup {\mathcal{X}}_3)-r({\mathcal{X}}_1\cup {\mathcal{X}}_2\cup {\mathcal{X}}_4).\end{array}$$

(55)

It has been open for years whether there exists subspace rank inequalities that are not implied by Ingleton inequalities and Shannon inequalities. It was until recently that the question was finally answered. In [5], insufficiency of Ingleton inequality to characterise all subspace rank inequalities was proved. And in [23,24], new subspace rank inequalities not implied by Ingleton inequalities were explicitly constructed. In fact, the set of subspace rank inequalities for up to five variables have all been determined. However, the complete characterisation involving more than five variables is still missing. In the following, we will review some of the important results along this line of work.

Theorem 14

(Kinser [23]) Suppose $\mathcal{X}=\left\{X_1,\dots ,X_n\right\}$ and h is representable over $\mathcal{X}$. Then

$$\begin{array}{c}h\left(X_{1,2}\right)+h\left(X_{1,3,n}\right)+h\left(X_3\right)+\sum _{i=4}^n(h\left(X_i\right)+h\left(X_{2,i-1,i}\right))\\ \le h\left(X_{1,3}\right)+h\left(X_{1,n}\right)+h\left(X_{2,3}\right)+\sum _{i=4}^n(h\left(X_{2,i}\right)+h\left(X_{i-1,i}\right)).\end{array}$$

(56)

Or equivalently,

$$\begin{array}{c}\hfill I_h(X_2;X_3)\le I_h(X_1;X_2)+I_h(X_3;X_n|X_1)+\sum _{i=4}^n{I}_h(X_2;X_{i-1}|X_i).\end{array}$$

(57)

Theorem 15

(Dougherty et al. [24]) Suppose $\mathcal{X}=\{A,B,C_1,\dots ,C_n\}$ and h is representable over $\mathcal{X}$. Then

$$\begin{array}{c}\hfill (n-1)I_h(A;B)\le \sum _{i=1}^n{I}_h(A;B|C_i)+\sum _{i=1}^{n}h\left(C_i\right)-h\left(C_1,\dots ,C_n\right).\end{array}$$

(58)

Remark: In addition to the inequalities obtained in Theorem 15, the work [24] found all subspace rank inequalities in five variables (called DFZ inequalities) and many more other new inequalities in six variables.

Definition 9 (ϵ-truncation)

Let h be a polymatroid over $\mathcal{Y}$ and $0\le ϵ\le h\left(\mathcal{Y}\right)$. Define g as follows where

$$\begin{array}{cc}\hfill g\left(\alpha \right)& \triangleq min\left(h\right(\alpha ),h(Y)-ϵ),\forall \alpha \subseteq Y.\hfill \end{array}$$

(59)

Then g is called the ϵ-truncation of h.

Definition 10 (Truncation-preserving inequalities)

Let $\mathcal{Y}=\{Y_i,i\in {\mathcal{N}}_m\}$. A set of rank inequalities

$$\begin{array}{c}\hfill \left\{\sum _{\alpha \subseteq {\mathcal{N}}_m}{c}_{\alpha }^{\ell }H(Y_i,i\in \alpha )\ge 0,\ell \in \Delta \right\}\end{array}$$

(60)

is said to preserve truncation (or is truncation-preserving) if for any h satisfying all the inequalities in (60), its truncation also satisfies all the inequalities.

Proposition 4

(Chan et al. [5]) DFZ inequalities are truncation preserving.

Theorem 16

(Insufficiency of truncation preserving inequalities [5]) Let ${\Delta }_n$ be the set of all subspace rank inequalities involving n variables (or subspaces). Then for sufficiently large n, ${\Delta }_n$ is not truncation-preserving.

5.2. Determinantal Inequalities

Information inequalities for Gaussian random variables are another interesting class of information inequalities. As we shall see, they are equivalent to determinantal inequalities.

Definition 11 (Gaussian polymatroid)

Let h be a polymatroid over $\mathcal{N}$. It is called Gaussian if there exists a set of jointly Gaussian random variables $\{Y_j,j\in \mathcal{K}\}$ with a $\left|\mathcal{K}\right|\times \left|\mathcal{K}\right|$ covariance matrix and a partition of $\mathcal{K}$ into n disjoint nonempty subsets ${\beta }_1,\dots ,{\beta }_n$ such that for any $\alpha \subseteq \mathcal{N}$,

$$\begin{array}{c}\hfill h\left(\alpha \right)=H\left(X_i,i\in \alpha \right).\end{array}$$

(61)

where $X_i=(Y_j,j\in {\beta }_i)$ for all $i\in \mathcal{N}$. Furthermore, h is called weakly Gaussian if there exists $\delta >0$ such that $\delta h$ is Gaussian, and almost Gaussian if h is the limit of a sequence of weakly Gaussian functions.

It is straightforward to prove that the weakly Gaussian property is closed under addition. In other words, if h and g are weakly Gaussian, then their sum $h+g$ is also weakly Gaussian. Furthermore, like information inequality for any continuous random variables, if an inequality

$$\begin{array}{c}\hfill \sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }H\left(X_{\alpha }\right)\ge 0\end{array}$$

(62)

holds for all Gaussian random variables $\{X_i,i\in \mathcal{N}\}$ [25], then it must be balanced. Therefore, in the following, we will only consider balanced information inequalities.

Let $\{Y_j,j\in \mathcal{K}\}$ be a set of jointly Gaussian random variables with covariance matrix K which is a $\left|\mathcal{K}\right|\times \left|\mathcal{K}\right|$ positive definite matrix. Suppose $\mathcal{K}$ is partitioned into n disjoint nonempty subsets ${\beta }_1,\dots ,{\beta }_n$. A very compelling property of a set of Gaussian random variable is that its entropy and the determinant of its covariance matrix is related by the following relation:

$$\begin{array}{c}\hfill H(Y_i,i\in \beta )=\frac{1}{2}log[{\left(2\pi e\right)}^{\left|\beta \right|}det\left(K_{\beta }\right)]=\left|\beta \right|\frac{log\left(2\pi e\right)}{2}+\frac{logdet\left(K_{\beta }\right)}{2}.\end{array}$$

(63)

where $K_{\beta }$ be the principal submatrix of K by deleting rows and columns that are not indexed by β. Substitute (63) back into (62), the inequality (62) is satisfied by all Gaussian random variables $\{X_i,i\in \mathcal{N}\}$ if and only if

$$\begin{array}{c}\hfill \sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }\left(|{\beta }_{\alpha }|\frac{log\left(2\pi e\right)}{2}+\frac{logdet\left(K_{{\beta }_{\alpha }}\right)}{2}\right)\ge 0\end{array}$$

(64)

where ${\beta }_{\alpha }={\bigcup }_{i\in \alpha }{\beta }_i$. Since the inequality (62) is balanced,

$$\begin{array}{c}\hfill \sum _{\alpha \subseteq \mathcal{N}:j\in \alpha }{c}_{\alpha }=0\end{array}$$

(65)

for all $j\in \mathcal{N}$. On the other hand,

$$\begin{array}{cc}\hfill \sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }\left|{\beta }_{\alpha }\right|& =\sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }\sum _{j\in \alpha }\left|{\beta }_j\right|\hfill \end{array}$$

(66)

$$\begin{array}{cc}& =\sum _{j\in \mathcal{N}}\sum _{\alpha \subseteq \mathcal{N}:j\in \alpha }{c}_{\alpha }\left|{\beta }_j\right|\hfill \end{array}$$

(67)

$$\begin{array}{cc}& =\sum _{j\in \mathcal{N}}\left|{\beta }_j\right|\sum _{\alpha \subseteq \mathcal{N}:j\in \alpha }{c}_{\alpha }\hfill \end{array}$$

(68)

$$\begin{array}{cc}& =0.\hfill \end{array}$$

(69)

Therefore, the inequality (62) holds for all Gaussian random variables if and only if the following determinantal inequality holds for all positive definite matrix K

$$\begin{array}{c}\hfill \sum _{\alpha \subseteq \mathcal{N}}{c}_{\alpha }logdet\left(K_{{\beta }_{\alpha }}\right)\ge 0\end{array}$$

(70)

or equivalently,

$$\begin{array}{c}\hfill \prod _{\alpha \subseteq \mathcal{N}}det{\left(K_{{\beta }_{\alpha }}\right)}^{c_{\alpha }}\ge 1.\end{array}$$

(71)

As a direct consequence, for any valid information inequality, we can use the above relation to derive a corresponding determinantal inequality. For example, the following well-known determinantal inequalities can all be proved using this “information-theoretical method”.

• (Hadamard inequality) Let K be a positive definite matrix K. Then

  $$\begin{array}{c}\hfill detK\le \prod _{i=1}^{\left|\mathcal{K}\right|}{K}_{i,i}\end{array}$$

  (72)

  where $K_{i,i}$ is the $i^{th}$ diagonal entry of K. This inequality follows from the following information inequality

  $$H\left(Y_1,\dots ,Y_k\right)\le \sum _{i=1}^{k}H\left(Y_i\right).$$
• (Szasz inequality) For any $1\le l<k$,

  $$\begin{array}{c}\hfill {\left(\prod _{\beta :\left|\beta \right|=l}det\left(K_{\beta }\right)\right)}^{1/\left(\genfrac{}{}{0pt}{}{k-1}{l-1}\right)}\ge {\left(\prod _{\beta :\left|\beta \right|=l+1}det\left(K_{\beta }\right)\right)}^{1/\left(\genfrac{}{}{0pt}{}{k-1}{l}\right)}.\end{array}$$

  (73)

  This determinantal inequality follows from the following information inequality

  $$\begin{array}{c}\hfill \frac{1}{\left(\genfrac{}{}{0pt}{}{k}{l}\right)}\sum _{\beta :\left|\beta \right|=l}\frac{H(Y_i,i\in \beta )}{l}\ge \frac{1}{\left(\genfrac{}{}{0pt}{}{k}{l+1}\right)}\sum _{\beta :\left|\beta \right|=l+1}\frac{H(Y_i,i\in \beta )}{l+1}.\end{array}$$

  (74)

Finally, we will conclude this section by the following open question: While Gaussian polymatroid is clearly almost entropic, is it true that an almost entropic polymatorid almost Gaussian? In other words, for any almost entropic polymatroid h, can we construct a sequence of Gaussian polymatroids $\{g_i,i=1,\dots \}$ such that

$$\underset{i\to \infty }{lim}{\delta }_i{g}_i=h$$

for some ${\delta }_i>0$ for all i.

6. Summary and Conclusions

In this paper, we have reviewed some of the recent progresses in characterisation of information inequalities. We first began with a geometric framework for information inequalities which has simplified the understanding of information inequalities. We also reviewed how the first non-Shannon inequality was proved and highlighted the general idea behind the proof. Next, we studied the infinite series of inequalities over ${\mathcal{N}}_4$ and considered a nonlinear relaxation of the series of inequalities.

We have also reviewed how information inequalities are related to Kolmogorov complexity inequalities, group-theoretic inequalities and inequalities for box assignments. Based on their relations, we demonstrated non-traditional approaches to proving information inequalities.

Finally, we investigated two constrained classes of information inequalities. The first class is when random variables are induced by vector spaces. In this case, the constrained inequalities are equivalent to subspace rank inequalities. We showed that Ingleton and DFZ inequalities are insufficient to characterise all subspace rank inequalities in general where the set of all subspace rank inequalities is not truncation-preserving. The second constrained class of inequalities is when random variables are Gaussian. We have showed that these constrained inequalities are in fact determinantal inequalities.

As a final remark, we would like to emphasise that this survey paper aims not to cover every aspect about information inequalities. In fact, there are many interesting pieces of work that we did not cover. For example, as pointed out by one of the reviewers, one very interesting area is about the relation between convex body inequalities and information inequalities [26,27]. We strongly encourage readers who are interested to further explore those relevant areas.