Algebraic Representations of Entropy and Fixed-Sign Information Quantities

Abstract

Many information-theoretic quantities have corresponding representations in terms of sets. Many of these information quantities do not have a fixed sign—for example, the co-information can be both positive and negative. In previous work, we presented a signed measure space for entropy where the smallest sets (called atoms) all have fixed signs. In the present work, we demonstrate that these atoms have natural algebraic behaviour which can be expressed in terms of ideals (characterised here as upper sets), and we show that this behaviour allows us to make bounding arguments and describe many fixed-sign information quantity expressions. As an application, we give an algebraic proof that the only completely synergistic system of three finite variables X, Y and $Z=f(X,Y)$ is the XOR gate.

1. Introduction

1.1. Information Decomposition

The Shannon entropy has many properties which intuitively mirror many set-theoretic identities. The I-measure of Yeung, built on earlier work by Hu Kuo Ting, fleshed out the correspondence between expressions of information quantities and set-theoretic expressions via a formal symbolic substitution [1,2]. Occasionally, however, the I-measure can be seen to conflate qualitatively different behaviours. A classic example of two such systems are the dyadic and triadic systems of James and Crutchfield [3], whose co-information signatures are identical, despite having qualitatively different constructions. In that work, they note that ‘no standard Shannon-like information measure, and exceedingly few nonstandard methods, can distinguish the two’.

One approach for discerning between these two systems is Partial Information Decomposition (PID), which aims to decompose the mutual information between a series of random source variables $X_1,\dots ,X_n$ and a target variable T into parts [4,5,6,7,8,9]. These parts are combinations of redundant, unique, and synergistic contributions to $I(X_1,\dots ,X_n;T)$—which are all qualitatively distinct. The last of these pieces, the synergistic information, is information that is provided by multiple sources considered together but no source alone. For example, the perception of depth is severely hindered unless both eyes are recruited, and hence depth information is conferred synergistically. The co-information between three variables is, under the PID framework, the redundant (shared) information minus the synergistic information. For this reason, negative co-information is evidence that the system is exhibiting synergistic behaviour.

Beloved amongst the practitioners of PID is the XOR gate—one of the few examples of synergy on which there is nearly unanimous agreement. We will show in this work that the XOR gate is in fact the only system of three variables $X,Y$ and $Z=f(X,Y)$ which can have purely synergistic behaviour.

To accomplish this, we will leverage our previously introduced refined signed measure space $\Delta \Omega$, representing a collection of ‘atomic’ pieces of information [10,11]. This decomposition, built from the entropy loss when merging variable outcomes into coarser events (see for instance [12] for more context), allowed us to construct a Shannon-like measure which can discern between the dyadic and triadic systems of James and Crutchfield [3], which was demonstrated in [11]. This decomposition, coupled with the investigation of its algebraic properties in the present work, leads us to the capstone XOR result in Theorem 9.

1.2. Contributions

In the present work, we demonstrate that the constructed space $\Delta \Omega$ has much natural algebraic behaviour when considered in tandem with the measure $\mu$. We show that the structure of co-information, a standard information quantity [13], can be expressed algebraically inside of $\Delta \Omega$, and this description has very stable behaviour under the measure $\mu$.

In Section 2, we present some background on the measure $\mu$ and recapitulate the main concepts introduced in [11]. From there, in Section 3, we develop the algebraic theory of this decomposition, introducing a new object, the ideal, and we show that it has some natural properties for simplifying representations of subsets in $\Delta \Omega$, highlighting how they can be used to generalise partitions in $\Omega$.

After this, in Section 4, we show that the algebraic structure of these ideals plays uniquely well with the measure $\mu$, allowing us to describe an algebraic property we call ‘strong fixed parity’, which, we show, corresponds to an information quantity having a fixed sign.

Lastly, in Section 5, we use these ideas to investigate the co-information between systems of variables, showing that this can be demonstrably fixed-sign in many cases. We finish with a result showing that the XOR gate is the only purely synergistic deterministic gate in three variables. That is, given three discrete variables X and Y with $Z=f(X,Y)$ where X and Y are finite, the XOR is the only system with negative co-information for all probability mass functions on X and Y.

To start, we give a brief recapitulation of the concepts introduced in the previous work [11] here as background. Proofs for all new results can be found in Appendix A.

2. Background

2.1. Background on the Measure

In previous work, we introduced the signed measure space $(\Delta \Omega ,\mu )$, where the $\sigma$-algebra is taken implicitly as the set of all subsets of $\Delta \Omega$. To start, we restate the definition of the space $\Delta \Omega$ as was given in the first paper [10].

Definition 1. 

Let $(\Omega ,\mathcal{F},P)$ be a finite probability space where the σ-algebra $\mathcal{F}$ is given by all subsets of Ω. Then, we define the  complex  (or content) of Ω, written $\Delta \Omega$, to be the simplicial complex on all outcomes $\omega \in \Omega$, with the vertices removed:

$$\Delta \Omega =\bigcup _{k=2}^N{\Omega }_k\cong \mathcal{P}(\Omega )\setminus \left(\left\{\right\{\omega \}:\omega \in \Omega \}\cup \{⌀\}\right)$$

(1)

where ${\Omega }_k$ is the set of all subsets of size k inside of Ω.

This space contains $2^{|\Omega |}-|\Omega |-1$ elements (called atoms) for a given finite outcome space $\Omega$.

Definition 2. 

Given a discrete outcome space Ω, an  atom  is a subset $S\subseteq \Omega$ where $\left|S\right|\ge 2$.

For a general set S, we use the notation $b_S$ for an atom, but where outcomes are explicitly labelled, e.g., $S=\{1,2,3\}$, we might write $b_{\{1,2,3\}}$, $b_{123}$ or simply 123 where this is clear from context.

In order to construct the signed measure space $(\Delta \Omega ,\mu )$, we must also define the measure. In the original work [10,11], this representation of the measure is given as a proposition. We give it here as the primary definition.

Definition 3. 

Let $T=\{p_1,\dots ,p_k\}$ be some subset of probabilities of an atom $\{{\omega }_1,\dots ,{\omega }_n\}$. For clarity, we write

$$\sigma \left(T\right)=\sigma (p_1,\dots ,p_k)={(p_1+\cdots +p_k)}^{(p_1+\cdots +p_k)}.$$

(2)

Taking all subsets of the atom $\{{\omega }_1,\dots ,{\omega }_n\}$ of size k, we write

$$A_k=\prod _{\begin{array}{c}S\subseteq \{p_1,\dots ,p_n\}\\ \left|S\right|=k\end{array}}\sigma \left(S\right).$$

(3)

Then the measure on the atom is given by

$$\mu (p_1,\dots ,p_n)=\sum _{k=1}^n{(-1)}^{n-k}log\left(A_k\right).$$

(4)

This definition arises from the perspective of entropy loss, which has appeared previously in the literature and has some natural advantages over the classical formulation of entropy [12]. The measure given here is constructed using two steps: firstly, by considering the entropy loss L when a number of outcomes ${\omega }_1,\dots ,{\omega }_t$ are merged and treated as a single outcome; and secondly, by performing a Möbius inversion with L over the partially ordered set of subsets of $\Omega$ (ordered under inclusion).

Using the measure of loss L alone, while sufficient to derive a measure space (see [14]), does not possess sufficient resolution to capture all information quantities, missing quantities such as the mutual information and co-information. Incorporating the Möbius inversion breaks the construction into smaller pieces which multiple systems might share, creating an additive measure $\mu$.

The entropy loss, as given immediately by a result of Baez et al. [12], is homogeneous of degree d when applied to the d-th Tsallis entropy [15]. By extension, the measure $\mu$, which can be viewed as an alternating sum of the losses L, is also homogeneous of degree d when built on the d-th order Tsallis entropy. Moreover, the measure $\mu$ has some intriguing properties, which we shall briefly restate here. The interested reader should refer to the original works [10,11] for more detail.

Example 1. 

Let $\Omega =\{{\omega }_1,{\omega }_2,{\omega }_3,{\omega }_4\}$ with corresponding probabilities $p_1=0.1$, $p_2=0.2$, $p_3=0.3$ and $p_4=0.4$. The atom $b_{12}$, which we might also write simply as 12 or $\{1,2\}$, has corresponding measure

$$\mu ({\omega }_1,{\omega }_2)=\mu (0.1,0.2)=0.275\mathrm{bits}.$$

(5)

The atom $b_{123}$, meanwhile, has a negative sign. Using the method given above, this is given by

$$\begin{array}{cc}\hfill \mu ({\omega }_1,{\omega }_2,{\omega }_3)& ={log}_2{\displaystyle \frac{{(0.1+0.2+0.3)}^{(0.1+0.2+0.3)}·0.1^{0.1}·0.2^{0.2}·0.3^{0.3}}{{(0.1+0.2)}^{(0.1+0.2)}·{(0.1+0.3)}^{(0.1+0.3)}·{(0.2+0.3)}^{(0.2+0.3)}}}\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill & ={log}_2{\displaystyle \frac{0.6^{0.6}·0.1^{0.1}·0.2^{0.2}·0.3^{0.3}}{0.3^{0.3}·0.4^{0.4}·0.5^{0.5}}}=-0.210\mathit{bits}.\hfill \end{array}$$

(7)

We will see in Theorem 1 that this change in sign is inevitable for certain atoms.

Lemma 1. 

For $p_1,\dots ,p_n,x\in {\mathbb{R}}^{+}$ where $n\ge 0$, we have

$$\underset{x\to 0}{lim}\mu (p_1,\dots ,p_n,x)=0.$$

(8)

This lemma guarantees that the measure becomes null if any of the constituent probabilities are zero.

Lemma 2. 

Let $p_1,\dots ,p_{n-1},x\in {\mathbb{R}}^{+}$ and let x vary. Then

$$\underset{x\to \infty }{lim}|\mu (p_1,\dots ,p_{n-1},x)|=|\mu (p_1,\dots ,p_{n-1})|.$$

(9)

This result shows that if one of the ‘probabilities’ tends to infinity, then the size of the entropy contribution tends towards that of an atom lying beneath it. Although discrete probabilities cannot tend to infinity, this result will prove useful for bounding arguments as part of Corollary 1 below.

Lastly, as a particularly intriguing property of the measure, its sign is known on all atoms of the partial order.

Theorem 1. 

Let $p_2,\dots ,p_n\in {\mathbb{R}}^{+}$ be a sequence of nonzero arguments for $n\ge 2$ and $m\ge 0$. Then

$${(-1)}^{m+n}{\displaystyle \frac{{\partial }^m\mu }{\partial x^m}}(x,p_2,\dots ,p_n)\ge 0.$$

(10)

Setting $m=0$, it becomes clear that the sign of the measure $\mu$ on a given atom ${\omega }_1,\dots ,{\omega }_n$ is dependent only on the number of outcomes n. The co-information, by contrast, is not a fixed-sign quantity in general. For example, given three random variables, the co-information can be positive, negative, or even dependent on the underlying probabilities.

Coupled with Lemma 2, we have that $\mu$ varies monotonically between 0 and the magnitude of the atoms beneath it.

Corollary 1 

(Magnitude can only decrease). Let $p_1,\dots ,p_{n-1},\tau \in {\mathbb{R}}^{+}\cup \left\{0\right\}$ for $n\ge 3$. Then

$$|\mu (p_1,\dots ,p_{n-1},\tau )|<|\mu (p_1,\dots ,p_{n-1})|.$$

(11)

This corollary is intriguing in that it bounds the contribution to the entropy of an atom by all of the atoms which lie under it in the partial order. This can be thought of as the notion that ‘higher-order contributions to the entropy are bounded above by lower-order contributions to the entropy’.

2.2. Ideals in Ring Theory

It may be helpful for some readers to briefly introduce the notion of an ideal as it appears in the algebraic theory of rings, since we will introduce an analogous object in the next section. While the ideals introduced in this work are constructs inside of a lattice (rather than a ring), they are usually first introduced inside of rings, where their structure is intuitive. In addition, there are ways in which it might be natural to extend the definition given in the remainder of this work to an ideal in a ring. Thus, we have chosen to use the name ‘ideal’ rather than ‘order ideal’ (as might be more standard). The reader familiar with the algebraic theory of rings and ideals can confidently skip this subsection.

A ring is, broadly speaking, a set where there exist notions of addition, subtraction, and multiplication (though not division, in general). A standard example is the integers $\mathbb{Z}$ or the ring of polynomials in a single variable x with real coefficients $\mathbb{R}\left[x\right]$.

Definition 4 

(Ideal in a ring). An ideal I over a (commutative) ring R is a subset $I\subseteq R$ such that I is a group under addition inside of I and closed under multiplication with an element of R. That is, for any $x,y\in I$ and $r\in R$, we have

$$\begin{array}{cc}\hfill -x& \in I\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill x+y& \in I\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill rx& \in I.\hfill \end{array}$$

(14)

Note that because of the first and second requirements, every ring ideal also contains zero.

Ideals capture a notion of dependency between elements in the ring. The presence of one element in the ideal forces those ‘above’ the element to also be contained in the ideal (where the order can be described by multiplication/divisibility). Ideals also have some convenient properties.

Proposition 1. 

Let $I,J$ be two ideals of a ring R. Then

$$\begin{array}{cc}\hfill I\cap J& =\{x\in R:x\in I\mathrm{and}x\in J\}\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill I+J& =\{x+y:x\in I,y\in J\}\hfill \end{array}$$

(16)

are both ideals.

A classic example of an ideal in $\mathbb{Z}$ is $\langle n\rangle$, which is the set of all numbers which are divisible by an integer n. If a and b are elements of $\langle n\rangle$ (i.e., they are both divisible by n), then we certainly must have that $a+b\in \langle n\rangle$ (their sum is also divisible by n), and multiplying by any number $r\in \mathbb{Z}$ will force $ar\in \langle n\rangle$, as the factor of n is still present.

Ideals also play a large role in algebraic geometry, where polynomial rings are a natural point of study. In this scenario, the ideal $\langle f\left(x\right)\rangle \subseteq \mathbb{R}\left[x\right]$ is the set of all polynomials which contain $f\left(x\right)$ as a factor. Equivalently, it is the set of polynomials which, given that $f\left(x\right)=0$, must also be zero.

In much the same way that knowledge that n is divisible by 2 implies $3n$ is divisible by 2, knowledge that two outcomes ${\omega }_1,{\omega }_2$ are distinct automatically provides knowledge that some pair inside of ${\omega }_1,{\omega }_2,{\omega }_3$ is distinct. This is the structure of dependency which we make use of when restating Definition 5 below.

3. An Algebraic Perspective on Entropy

3.1. Representing Information Quantities Inside $\Delta \Omega$

We briefly state a key result from our previous work [11], where we expressed the entropy associated to a random variable X in terms of a subset of $\Delta \Omega$.

Definition 5. 

Given a random variable X, we define the  content  $\Delta X$ inside of $\Delta \Omega$ to be the set of all atoms inside of $\Delta \Omega$ crossing a boundary in X. That is, if X corresponds to a partition $P_1,\dots ,P_n$, then

$$\Delta X=\{b_S:S\subseteq \Omega ,\exists {\omega }_i,{\omega }_j\in S\mathit{with}{\omega }_i\in P_k,{\omega }_j\in P_l\mathit{such}\mathit{that}k\ne l\}.$$

(17)

Intuitively, this means that at least two of the outcomes in the atom $b_{{\omega }_1\dots {\omega }_n}$ correspond to distinct events in X, although possibly more. We will in general make use of Δ to represent the logarithmic decomposition functor from random variables and information quantities to their corresponding sets in $\Delta \Omega$. Note that we often write 123 to refer to $b_{1,2,3}$ for added readability.

As expected, we have that $\mu (\Delta X)=H\left(X\right)$, and we concretise this in a theorem, which is taken from [11].

Theorem 2. 

Let R be a region on an I-diagram of variables $X_1,\dots ,X_r$ with Yeung’s I-measure. In particular, R is given by some set-theoretic expression in terms of the set variables ${\tilde{X}}_1,\dots ,{\tilde{X}}_r$ under some combination of unions, intersections and set differences.

Making the formal substitution

$${\tilde{X}}_1,{\tilde{X}}_2,\dots ,{\tilde{X}}_r⟷\Delta X_1,\Delta X_2,\dots ,\Delta X_r$$

(18)

to obtain an expression $\Delta R$, the content corresponding to the region R of the I-diagram, in terms of the $\Delta X_i$, we have

$$I\left(R\right)=\sum _{B\in \Delta R}\mu \left(B\right).$$

(19)

That is, the interior loss measure μ is consistent with Yeung’s I-measure.

For examples on how this measure can be interpreted geometrically, as well as all proofs of the above results, we refer the interested reader to the previous work [11], where we present figures and diagrams demonstrating the geometric and set-theoretic significance of the atoms of our construction.

Many questions about the underlying structure of this space remain to be answered. One peculiarity is that most atoms do not normally appear alone in information quantities. For example, given that an atom ${\omega }_1\dots {\omega }_n\in \Delta X$ appears in a content, we must also have that the atom ${\omega }_1\dots {\omega }_n{\omega }_{n+1}\in \Delta X$ appears in the same content, as the definition is just those atoms which, as a set, cross a boundary in X. While all atoms have an interpretation of crossing boundaries in partitions, individual atoms, at first, do not seem to have much meaning without other atoms in context. Understanding the structural interrelationship between all atoms would allow for a better understanding of the relationship between different information measures.

In the rest of this section, we explore the structure of our decomposition in the language of posets and upper sets (or ideals) on those posets, which appear to provide the natural language for the analogous ‘molecules’ to our atoms. We begin by defining an order ≼ on our atoms before giving a definition for ideals in $\Delta \Omega$. From there, we will show that all co-information expressions correspond to ideals and vice versa. We finish this section by characterising the ideals which correspond to the entropy of a variable.

3.2. Ideals in $\Delta \Omega$

Definition 6. 

Let $b_{S_1},b_{S_2}\in \Delta \Omega$ where $S_1,S_2\subseteq \Omega$. We define a partial ordering on the set $\Delta \Omega$ by setting $b_{S_1}\preccurlyeq b_{S_2}$ whenever $S_1\subseteq S_2$.

The following definition is taken from [16].

Definition 7. 

Given a (partially) ordered set P, a subset $J\subseteq P$ is called an  order ideal  (upper-set,  up-set,  increasing set) if, for all $x\in J$ and $y\in P$, we must have $y\in J$ whenever $x\preccurlyeq y$ and $J\ne ⌀$. That is, J is non-empty and closed under ascending order.

Following standard language, we will say that an ideal J is  generated  by a collection of elements $g_1,\dots ,g_t$ if, for all $b\in J$, we have $g_i\preccurlyeq b$ for at least one $g_i\in \{g_1,\dots ,g_t\}$. We will write $J=\langle g_1,\dots ,g_t\rangle$.

That is to say, the ideal J is the set in $\Delta \Omega$ which contains $g_1,\dots ,g_t$ and all elements which lie above them in the order.

We note that we deviate from standard nomenclature in this case and refer to these upper sets simply as ‘ideals’. In classical order theory, ideals in lattices are down sets rather than upper sets and are subject to an additional constraint. In the current work, we use ideal in the order-ideal sense, as we expect that future work on $\Delta \Omega$ as a ring might make this definition more intuitive.

We will concern ourselves later with the relationship between the generators of an ideal and the measure of the ideal itself. The following definition for the degree of an atom is taken from [11], which we then extend to ideals.

Definition 8. 

Let $b={\omega }_1\dots {\omega }_d\in \Delta \Omega$. We define the  degree  of b to be the number of outcomes it contains. That is, $deg\left(b\right)=d$.

Definition 9. 

We will call J a  degree n ideal  or  n-ideal  if it can be generated by purely degree n atoms.

One significant motivation for introducing the language of ideals is to simplify the description of the sets constructed by the decomposition. Rather than writing out the complete set of all atoms, it is often possible to write out the generators of the set as an ideal, vastly reducing the complexity of the notation. Much like ideals in ring theory, it is straightforward to describe the intersection and union of ideals using the generators alone. We introduce some notation and give a proposition to this effect.

Notation 1. 

To further the parallel to ideals in rings, it is sometimes useful to introduce multiplicative notation for the union of two atoms. That is, we will make use of the notation

$$b_{S\cup T}=b_S{b}_T.$$

(20)

For example, using the shorthand notation from before, we have $123·234=1234$.

Proposition 2. 

Let $G=\{g_1,\dots ,g_n\}$ be a set of generators for the ideal $I=\langle G\rangle =\langle g_1,\dots ,g_n\rangle$, and let $H=\{h_1,\dots ,h_m\}$ be a set of generators for the ideal $J=\langle H\rangle =\langle h_1,\dots ,h_m\rangle$, where I and J are ideals inside $\Delta \Omega$. Then,

$$I\cup J=\langle g_1,\dots ,g_n,h_1,\dots ,h_m\rangle =\langle b\mid b\in G\cup H\rangle ,$$

(21)

$$I\cap J=\langle g_1{h}_1,g_1{h}_2,\dots ,g_n{h}_{m-1},g_n{h}_m\rangle =\langle gh\mid g\in G,h\in H\rangle$$

(22)

This formulation mimics the natural behavior of ideals in rings.

Remark 1. 

It is occasionally convenient for notation to consider ideals generated by single outcomes ω even though we formerly excluded these and ⌀ from $\Delta \Omega$. We may alternate between including and excluding these atoms for algebraic simplicity. Recall that the singlet and empty atoms do not contribute to the entropy, so this choice of notation does not affect the measure (we note also that including these entities would endow $\Delta \Omega$ with the complete structure of a lattice, which, although currently not required, might be useful for future work).

Example 2. 

Let $\Omega =\{1,2,3,4\}$. Then $\Delta \Omega =\{12,13,14,23,24,34,123,124,134,234,1234\}$. Inside of $\Delta \Omega$, an ideal consists of all atoms which ‘contain’ a generator. For example, the ideal $\langle 12,13\rangle =\{12,13,123,124,134,1234\}$ is a 2-atom ideal, as it is generated by degree 2 atoms. All of the atoms in this ideal must contain either outcomes 1 and 2 or outcomes 1 and 3, or both.

3.3. Representation of Quantities with Ideals

We should justify that these ideals are a natural object of study. As it turns out, all entropy expressions without multiplicity and without conditioning are given by ideals, which follows from the next lemma.

Lemma 3. 

Given any co-information $I(X_1;\dots ;X_t)$ (including entropy and mutual information), the corresponding content $\Delta X_1\cap \cdots \cap \Delta X_t$ is an ideal.

Example 3. 

It is worth noting that ideals themselves do not, in general, have corresponding partitions, but every partition has a corresponding ideal. As an example for how to conceptualise these ‘sub-partitions,’ consider the system $\Omega =\{1,2,3\}$ where X has partition $\left\{\right\{1\},\{2,3\left\}\right\}$ and Y has partition $\left\{\right\{1,3\},\{2\left\}\right\}$ as per Figure 1.

We have that

$$\begin{array}{cc}\hfill \Delta X& =\{12,13,123\}\mathit{and}\hfill \end{array}$$

(23)

$$\begin{array}{cc}\hfill \Delta Y& =\{12,23,123\}.\hfill \end{array}$$

(24)

Taking the mutual information between these sets corresponds algebraically to the intersection $\Delta X\cap \Delta Y=\langle 12,13\rangle \cap \langle 12,23\rangle =\langle 12\rangle =\{12,123\}$.

We note that this upper set $\langle 12\rangle$ corresponds to the ability to discern between 1 and 2, but not between 1 and 3, or 2 and 3. Moreover, the upper set $\langle 12\rangle$, despite not representing a partition itself, gives the mutual information when measured, i.e., $\mu (\langle 12\rangle )=I(X;Y)$.

That is to say, generalising from the language of partitions to the language of ideals has allowed us to properly describe mutual information—a quantity which partitions cannot in general represent.

As it turns out, the converse to Lemma 3 also holds, which we state here.

Theorem 3. 

Let Ω be a finite outcome space and let $\{X_a:a\in A\}$ be the collection of all possible random variables defined on Ω (indexed by A). Then, there is a one-to-one correspondence

$$\left\{\mathit{ideals}\mathit{in}\Delta \Omega \right\}\leftrightarrow \{\mathit{possible}\mathit{co}\text{-}\mathit{informations}\mathit{on}\Omega \}$$

(25)

where the co-informations are given by $I(X_1;\dots ;X_j)$ for any number of arbitrary variables $X_1,\dots ,X_j$ defined on Ω.

This result tells us that for any valid co-information on some collection of variables defined on an outcome space $\Omega$, then there is a corresponding ideal in $\Delta \Omega$, and for every ideal in $\Delta \Omega$, there is a corresponding collection of variables which give the resulting co-information. As an immediate side effect of this result, we have an alternative derivation of proposition 33 in [11]:

Corollary 2. 

Let Ω be a finite outcome space. Then, there is a one-to-one correspondence

$$\left\{\mathit{subsets}\mathit{of}\Delta \Omega \right\}\leftrightarrow \left\{\mathit{entropy}\mathit{expressions}\mathit{without}\mathit{multiplicity}\mathit{on}\Omega \right\},$$

(26)

where by an ‘entropy expression without multiplicity’ we mean an expression of the form

$$\sum _{P\mathit{partitioning}\Omega }{n}_{P}H\left(P\right)$$

(27)

for $n_P\in \mathbb{Z}$ where no region is double-counted in any I-diagram.

Note that for the purposes of these two results, we do not consider the singlet atoms $\left\{\omega \right\}$ or the empty set to be elements of $\Delta \Omega$, as they contribute no entropy (see [11] for more justification).

This result shows that with clever inclusion and exclusion, it is always possible to extract individual atoms as classical entropy expressions on variables in $\Delta \Omega$. That is, they form a natural basis for entropy expressions. As such, the atoms of $\Delta \Omega$ are uniquely placed for a module-theoretic or vector-space perspective on information.

Since these atoms appear to be a natural basis for entropy expressions, if we count them without multiplicity, we are able to determine how many expressions for information exist without accounting for the same contribution multiple times. We give a corollary to this end.

Corollary 3. 

Given a finite outcome space Ω with $|\Omega |=n$, there are $2^{2^n-n-1}$ possible classical entropy expressions without multiplicity.

Counting with multiplicity, we can see that the space of all entropy expressions on $\Omega$ is a free module over $\mathbb{Z}$, where the atoms b form a very natural basis.

We now state a practical result which tells us, intuitively, exactly which ideals correspond to partitions and how we can find the generators of the ideal corresponding to a finite random variable X. For the purpose of this result, it is again useful to consider the singlets $\left\{\omega \right\}$, but the resulting representation of $\Delta X$ will not contain them.

Theorem 4. 

Let X be a discrete random variable on the outcome space Ω, where X has corresponding partition $Q_t:t\in T$ for some indexing set T of parts $Q_t$. Then $\Delta X$ as an ideal is given by

$$\Delta X=\bigcup _{\begin{array}{c}a,b\in T\\ a\ne b\end{array}}\langle \{\omega \in Q_a\}\rangle \cap \langle \{\omega \in Q_b\}\rangle .$$

(28)

We note in particular that in posets, the union of order ideals is equal to the order ideal with the union of their generators. Equivalently, we have

$$\Delta X=\bigcup _{t\in T}\langle \{\omega \in Q_t\}\rangle \cap \langle \{\omega \in Q_t^c\}\rangle =\bigcup _{t\in T}\langle \{\omega \overline{\omega }:\omega \in Q_t,\overline{\omega }\in Q_t^c\}\rangle .$$

(29)

In particular, $\Delta X$ as an ideal is generated by 2-atoms.

Example 4. 

Consider the outcome space $\Omega =\{1,2,3,4\}$. Now, let X be the variable with partition $\left\{\right\{1,2\},\{3\},\{4\left\}\right\}$. Then, as an ideal, we have

$$\begin{array}{cc}\hfill \Delta X& =(\langle 1,2\rangle \cap \langle 3\rangle )\cup (\langle 1,2\rangle \cap \langle 4\rangle )\cup (\langle 3\rangle \cap \langle 4\rangle )\hfill \\ \hfill & =\langle 13,23\rangle \cup \langle 14,24\rangle \cup \langle 34\rangle \hfill \\ \hfill & =\langle 13,23,14,24,34\rangle .\hfill \end{array}$$

(30)

Corollary 4. 

Let $X_a,a\in A$ be a family of discrete variables on the outcome space Ω. Knowledge of how the 2-atoms are located among the $\Delta X_a$ is sufficient to describe how all other atoms are located.

Restating this, knowledge of the 2-atoms contained in each $\Delta X_a$ is sufficient to deduce the presence of any atom in any set-theoretic expression constructed using the $\Delta X_a$.

We have now successfully described the structure of entropy through the algebraic lens of ideals in a poset and illustrated that ideals in this lattice correspond to co-informations, while other subsets of $\Delta \Omega$ correspond to entropy expressions on $\Omega$.

To illuminate the power of this flavour of the theory, in the next section, we shall see how these ideals interact with the measure $\mu$ and use our results to demonstrate that mutual information is always given by a degree 2 ideal. Not only this, but we will give a generalisation which bounds the degree of the generators for ideals representing the intersection of more than two variables. We then extend these techniques to explore ideals giving fixed-sign information quantities. This intriguing result will show that a surprising amount can be learned about an information quantity without much knowledge of the underlying probabilities.

4. Properties of the Measure on Ideals

We have now developed lots of language for discussing the ideals inside of the lattice $\Delta \Omega$. Moreover, having seen that co-information is perfectly described by these ideals, it would be a natural question to ask how the measure $\mu$ interacts with the ideal structure. In this section, we will demonstrate that the entropy contribution of an ideal can, much like atoms, be neatly categorised as either positive or negative in many cases, and we shall see that this provides various tools for constructing new bounds.

In this section, we begin by demonstrating that the mutual information is always given by a degree 2 ideal. To accomplish this, we shall need the following notion of restriction, which we shall utilise in the proofs to follow.

Definition 10. 

Let X be a random variable on a finite outcome space Ω, and let $S\subseteq \Omega$. We define the  restriction  of a collection of atoms $W\subseteq \Delta \Omega$ to S as

$$W_S=\{b_Q\in W:Q\subseteq S\}.$$

(31)

In particular, we will use the notation $\Delta X_S$ and ${\langle \dots \rangle }_S$ or occasionally ${·|}_S$ to construct contents and ideals inside of restrictions.

Restriction simply allows us to focus our attention on a subset of the atoms—in particular, those whose outcomes all belong to the restricting subset S. Note that given some subset $W\subseteq \Delta \Omega$, we have that $W_S\subseteq W$ rather than operating with an entirely new class of atoms.

One of the strengths of the measure $\mu$ beyond entropy alone is that $\mu$ is homogeneous and works across multiple scales. As such, every statement and piece of structure given here for $\Delta \Omega$ and ideals therein also applies to $\Delta {\Omega }_S$ for $S\subseteq \Omega$. We shall demonstrate that many problems exploring the intersection of ideals (and hence the intersection of entropies) can be much simplified by restricting. We proceed with the first result on mutual information, where we use this concept in the proof.

Theorem 5 

(Mutual information is a degree 2 ideal). Let X and Y be two random variables. Then, there exists a set of 2-atom generators $\{a_i{b}_i:a_i,b_i\in \Omega \}$ for $i=1,\dots ,k$ such that

$$I(X;Y)=\mu \left(\langle a_1{b}_1,\dots ,a_k{b}_k\rangle \right)$$

(32)

We have demonstrated something rather intriguing: mutual information looks a lot like a normal variable content in that it is always generated by degree 2 atoms, but the generators of mutual information do not need to correspond to a representable subset of $\Delta \Omega$. When working with ideals in general, one would expect that the intersection between generators of degree m and n would have degree bounded above by $m+n$, so it is rather surprising that the mutual information has this property.

Extending the investigation of the Gács–Körner common information in [11], the following can now be seen:

Corollary 5. 

The Gács-Körner common information is generated by degree 2 atoms, and the generating set is the largest subset of generators of the mutual information, which is representable by some random variable (in [10] this property is referred to as ‘discernibility’).

This result confirms our natural intuition for selecting generators to construct a variable. We provide also a generalisation of Theorem 5 to co-information.

Theorem 6. 

Let Ω be the joint outcome space of M discrete variables $X_1,\dots ,X_M$. Then, the content of $I(X_1;\dots ;X_M)$ can be completely generated by atoms of degree at most M.

This result states that the degree of the generators of an ideal corresponding to some co-information is always bounded above by the number of variables. This result vastly reduces the search space of generators when studying the properties of co-information, and we make use of it in our study of fixed-parity systems in the next section.

Example 5. 

Consider the standard OR gate given by outcomes $(X,Y,Z=\mathrm{OR}(X,Y\left)\right)$, which we label as follows:

X	Y	$Z=\mathit{OR}(X,Y)$	Outcome $\left(\omega \right)$
0	0	0	1
0	1	1	2
1	0	1	3
1	1	1	4

Note that in this instance, we have

$$I(X;Y;Z)=\mu (\langle 14,123\rangle ),$$

(33)

which is generated by at most degree 3 atoms, as expected. Note that we are discussing the structure of the OR gate without any mention of the probabilities. A diagram representing the structure is given in Figure 2. As expected, the degree of the generators is bounded above by 3.

Although this is an interesting representation of the structure of the co-information between random variables, we have not said much yet about the relationship between these ideals and their measures $\mu \left(J\right)$. As it turns out, for certain classes of ideals, the sign of $\mu \left(J\right)$ is just as easy to characterise as the signs of the atoms themselves.

Lemma 4. 

Let $J=\langle {\omega }_1\cdots {\omega }_d\rangle$ be an ideal generated by a single degree d atom with  $\mathbb{P}\left({\omega }_1\right),\dots ,\mathbb{P}\left({\omega }_d\right)\ne 0$. Then, ${(-1)}^d\mu \left(J\right)>0.$

This result is quite powerful, as it tells us that in certain scenarios, we can know the sign of the ideal and the information measure it represents without any knowledge of the probabilities. We will strengthen this result shortly to demonstrate that certain classes of ideals, which we call strongly fixed-parity ideals, have fixed-sign measures.

Definition 11. 

Let $J=\langle g_1,\dots ,g_j\rangle$ be an ideal. If $j=1$, we say that J is  strongly fixed parity, and set the  parity  of J as $P\left(J\right)={(-1)}^{deg{g}_1}.$

Moreover, if $j\ge 2$, we shall say that J has  strongly fixed parity  if there is an expression

$$\mu \left(J\right)=P\left(J\right)\sum _{\alpha \in A}P\left(J_{\alpha }\right)\mu \left(J_{\alpha }\right)$$

(34)

for some finite collection of fixed-parity ideals $\{J_{\alpha }:\alpha \in A\}$, with the equality holding across all probability distributions $\mathbb{P}$ on Ω and some $P\left(J\right)\in \{-1,1\}$, which we call the  parity  of J.

Lastly, we shall say that an ideal $J\subseteq \Delta \Omega$ is of  strongly mixed parity  if it has generators of both even and odd degree.

Example 6. 

The ideal $\langle 12,23\rangle$ is strongly fixed even parity as

$$\begin{array}{cc}\hfill \mu (\langle 12,23\rangle )& =\mu (\langle 12\rangle )+\mu (\langle 23\rangle )-\mu (\langle 12\rangle \cap \langle 23\rangle )\hfill \\ \hfill & =\mu (\langle 12\rangle )+\mu (\langle 23\rangle )-\mu (\langle 123\rangle ).\hfill \end{array}$$

(35)

The ideal $\langle 123\rangle$ has strong negative (odd) parity, and the two degree 2 ideals have strong positive (even) parity. When composing the parities, the measure of the whole set is given by three positive parts, so it makes sense to call $\langle 12,23\rangle$ positive fixed-parity.

Theorem 7 

(Ideals of Strong Parity). Let J be an ideal in $\Delta \Omega$. If J is of strongly even parity, then $\mu \left(J\right)\ge 0$, and if J is of strongly odd parity, then $\mu \left(J\right)\le 0$.

This result is most pleasant as it reflects what feels like a natural intuition for how these systems should behave. In particular, given a system of variables $X_1,\dots ,X_n$, defined by partitions on a finite outcome space $\Omega$, information quantities which reflect strongly fixed-parity ideals have a predetermined sign for any underlying probability distribution over $\Omega$.

In this section, we demonstrated that the algebraic construction of ideals in our poset from the previous section plays remarkably well with the measure $\mu$ of our construction, and we developed several tricks for manipulating expressions in $\Delta \Omega$. We build on this theory in the next section and apply it to the problem of finding purely synergistic systems of the form X, Y, and $Z=f(X,Y)$.

5. Fixed-Parity Systems

To motivate our investigation, we give here an example of how certain information quantities do not have fixed sign.

Example 7. 

Let X and Y be binary variables with $Z=\mathrm{OR}(X,Y)$.

Recall that the co-information (also known as the interaction information [13,17]) is given by

$$\begin{array}{cc}\hfill I(X;Y;Z)=& H\left(X\right)+H\left(Y\right)+H\left(Z\right)\hfill \\ \hfill & -H(X,Y)-H(X,Z)-H(Y,Z)\hfill \\ \hfill & +H(X,Y,Z).\hfill \end{array}$$

(36)

In the case that $P(X=x,Y=y)=0.25$ for all outcomes $(x,y)\in \Omega$, we have that the co-information is $I(X;Y;Z)\approx -0.19$ bits, being negative in this case. However, in the case that $P(X=0,Y=0)=P(X=1,Y=1)=0.45$ and $P(X=0,Y=1)=P(X=1,Y=0)=0.05$, then we have $I(X;Y;Z)\approx 0.52$ bits. That is, for this system (and many others), knowledge of the structure of the outcomes alone (that is, any prior knowledge that certain combinations of symbols have zero probability) is not sufficient to determine the sign of the co-information, and it depends upon the underlying probabilities of the system states. We shall see why this is the case for the OR gate in particular in this section.

We now give a definition to connect the algebraic perspective on ideals to information quantities.

Definition 12. 

Let $X_1,\dots ,X_n$ be a system of variables on a finite outcome space Ω. We say that the system $X_1,\dots ,X_n$ is a  fixed sign  or  fixed parity  system if the sign of

$$I(X_1;\dots ;X_n)$$

(37)

is fixed regardless of the underlying probability distribution. Similarly, we say that an entropy expression $E(X_1,\dots ,X_n)$ defined on Ω has  fixed parity  if its sign is always fixed regardless of the underlying probabilities in $X_1,\dots ,X_n$. We say that the system is  negative/odd fixed parity  if the co-information is always negative, and it is  positive/even fixed parity  if the co-information is always positive.

There is a natural dual question to be asked here: is it possible to have a fixed-parity system where the co-information is a strongly-mixed ideal—that is, generated by elements of varying degrees? The next theorem shows this is, in fact, impossible.

Theorem 8. 

A system of variables $X_1,\dots ,X_n$ with $I(X_1;\dots ;X_n)$ given by an ideal of strongly mixed-parity cannot have a fixed parity.

That is to say, being strongly-mixed (an algebraic property) implies mixed signs (a property of the measure on the set).

This theorem gives a partial converse to Theorem 7 in that it gives us a way to characterise some mixed-parity systems. Although we have not characterised every fixed-parity or mixed-parity system, we expect such a characterisation in terms of ideal properties (algebraic reasoning alone) should be possible. However, the tools we have already constructed are sufficient to show the main result of this paper.

Theorem 9. 

The only negative fixed-parity (always synergy-dominated) system given by two finite variables $X,Y$ and a deterministic function $Z=f(X,Y)$ is the XOR gate.

Of note is that the XOR gate is generated by the presence of three degree 3 atoms, as depicted in Figure 3. Each of these atoms generates the synergistic effect. While they all exhibit a change in X, Y and Z, the places where these changes are seen are distinct between the variables. There is no single degree 2 atom ${\omega }_1{\omega }_2$ (the knowledge that ${\omega }_1$ and ${\omega }_2$ are distinct) which the three variables share.

6. Discussion

In this paper, we extended our results on the measure $\mu$ from [10,11] to an algebraic construction inside of the space $\Delta \Omega$. We demonstrated that in many cases, the study of ‘ideals’ (in the order-theoretic sense) inside of $\Delta \Omega$ simplifies bounding problems, and we showed that these ideals form a natural intermediate language between partitions and have useful behaviour in tandem with the measure $\mu$.

While in the present work, the issue of bounding is applied to the study of fixed-sign quantities, we expect these techniques can be used in multiple scenarios where bounding over all possible probabilities is required. Moreover, we expect that there is a stronger version of Theorem 7 which might be stated with weaker restrictions on the underlying ideals, and a full characterisation of all fixed-parity systems would be an insightful direction. Future work may develop this theory.

One particularly intriguing result given here is that the underlying cause of three-variable synergy appears to be easily characterisable by its geometry alone. The three-outcome ‘flower’ shape presented in Figure 3 is required for the existence of synergy in three variables, being the only generator of three-variable co-information which has a negative measure, with an intrinsically three-dimensional shape. We leave here the problem of classifying the generators of various orders open. While it is straightforward to see all such effects now for two or three variables, the case beyond three variables appears to be more opaque. In particular, we expect that there may be vastly many new information effects in four or more variables which cannot geometrically exist in simpler systems.

We applied the new algebraic results developed in this paper to show that the XOR gate is the only purely synergistic system (i.e., always possessing negative co-information) of finite variables $X,Y$ and $Z=f(X,Y)$ for a deterministic function f (see also [18]). In particular, we highlight that this was achieved with a purely algebraic proof that did not require any navigation of the space of probability distributions.

We hope this work might be applied to the problem of Partial Information Decomposition (PID) [3,5,6,8,19] and contribute to the widening body of knowledge in set-theoretic information theory.