Intersection Information based on Common Randomness

The introduction of the partial information decomposition generated a flurry of proposals for defining an intersection information that quantifies how much of"the same information"two or more random variables specify about a target random variable. As of yet, none is wholly satisfactory. A palatable measure of intersection information would provide a principled way to quantify slippery concepts, such as synergy. Here, we introduce an intersection information measure based on the G\'acs-K\"orner common random variable that is the first to satisfy the coveted target monotonicity property. Our measure is imperfect, too, and we suggest directions for improvement.


Introduction
Partial information decomposition (PID) [1] is an immensely suggestive framework for deepening our understanding of multivariate interactions, particularly our understanding of informational redundancy and synergy.In general, one seeks a decomposition of the mutual information that n predictors X 1 , . . ., X n convey about a target random variable, Y .The intersection information is a function that calculates the information that every predictor conveys about the target random variable; the name draws an analogy with intersections in set theory.An anti-chain lattice of redundant, unique and synergistic partial information is then built from the intersection information.
As an intersection information measure, [1] proposes the quantity: I min (X 1 , . . ., X n : Y ) = y∈Y Pr(y) min i∈{1,...,n} D KL Pr X i |y Pr(X i ) , (1) where D KL is the Kullback-Leibler divergence.Although I min is a plausible choice for the intersection information, it has several counterintuitive properties that make it unappealing [2].
In particular, I min is not sensitive to the possibility that differing predictors, X i and X j , can reduce uncertainty about Y in nonequivalent ways.Moreover, the min operator effectively treats all uncertainty reductions as the same, causing it to overestimate the ideal intersection information.The search for an improved intersection information measure ensued and continued through [3][4][5], and today, a widely accepted intersection information measure remains undiscovered.
Here, we do not definitively solve this problem, but explore a candidate intersection information based on the so-called common random variable [6].Whereas Shannon mutual information is relevant to communication channels with arbitrarily small error, the entropy of the common random variable (also known as the zero-error information) is relevant to communication channels without error [7].We begin by proposing a measure of intersection information for the simpler zero-error information case.This is useful in and of itself, because it provides a template for exploring intersection information measures.Then, we modify our proposal, adapting it to the Shannon mutual information case.
The next section introduces several definitions, some notation and a necessary lemma.We extend and clarify the desired properties for intersection information.Section 3 introduces zero-error information and its intersection information measure.Section 4 uses the same methodology to produce a novel candidate for the Shannon intersection information.Section 5 shows the successes and shortcoming of our candidate intersection information measure using example circuits and diagnoses the shortcoming's origin.Section 6 discusses the negative values of the resulting synergy measure and identifies its origin.Section 7 summarizes our progress towards the ideal intersection information measure and suggests directions for improvement.Appendices are devoted to technical lemmas and their proofs, to which we refer in the main text.

Informational Partial Order and Equivalence
We assume an underlying probability space on which we define random variables denoted by capital letters (e.g., X, Y and Z).In this paper, we consider only random variables taking values on finite spaces.Given random variables X and Y , we write X Y to signify that there exists a measurable function, f , such that X = f (Y ) almost surely (i.e., with probability one).In this case, following the terminology in [8], we say that X is informationally poorer than Y ; this induces a partial order on the set of random variables.Similarly, we write X Y if Y X, in which case we say X is informationally richer than Y .
If X and Y are such that X Y and X Y , then we write X ∼ = Y .In this case, again following [8], we say that X and Y are informationally equivalent.In other words, X ∼ = Y if and only if one can relabel the values of X to obtain a random value that is equal to Y almost surely and vice versa.This "information-equivalence" can easily be shown to be an equivalence relation, and it partitions the set of all random variables into disjoint equivalence classes.The ordering is invariant within these equivalence classes in the following sense.If X Y and Y ∼ = Z, then X Z. Similarly, if X Y and X ∼ = Z, then Z Y .Moreover, within each equivalence class, the entropy is invariant, as shown in Section 2.2.

Information Lattice
Next, we follow [8] and consider the join and meet operators.These operators were defined for information elements, which are σ-algebras or, equivalently, equivalence classes of random variables.We deviate from [8] slightly and define the join and meet operators for random variables.
Given random variables X and Y , we define X Y (called the join of X and Y ) to be an informationally poorest ("smallest" in the sense of the partial order ) random variable, such that X X Y and Y X Y .In other words, if Z is such that X Z and Y Z, then Note that X Y is unique only up to equivalence with respect to ∼ =.In other words, X Y does not define a specific, unique random variable.Nonetheless, standard information-theoretic quantities are invariant over the set of random variables satisfying the condition specified above.For example, the entropy of X Y is invariant over the entire equivalence class of random variables satisfying the condition above.Similarly, the inequality Z X Y does not depend on the specific random variable chosen, as long as it satisfies the condition above.Note, the pair (X, Y ) is an instance of X Y .
In a similar vein, given random variables X and Y , we define X Y (called the meet of X and Y ) to be an informationally richest random variable ("largest" in the sense of ), such that X Y X and X Y Y .In other words, if Z is such that Z X and Z Y , then Z X Y .Following [6], we also call X Y the common random variable of X and Y .
An algorithm for computing an instance of the common random variable between two random variables is provided in [7]; it generalizes straightforwardly to n random variables.One can also take intersections of the σ-algebras generated by the random variables that define the meet.
The and operators satisfy the algebraic properties of a lattice [8].In particular, the following hold: Finally, the partial order is preserved under and , i.e., if X Y , then X Z Y Z and X Z Y Z.
Let H(•) represent the entropy function and H •|• the conditional entropy.We denote the Shannon mutual information between X and Y by I(X : Y ).The following results highlight the invariance and monotonicity of the entropy and conditional entropy functions with respect to ∼ = and [8].Given that X Y if and only if X = f (Y ), these results are familiar in information theory, but are restated here using the current notation: (c) X Y if and only if H(X|Y ) = 0.

Desired Properties of Intersection Information
We denote I (X : Y ) as a nonnegative measure of information between X and Y .For example, I could be the Shannon mutual information; i.e., I (X : Y ) ≡ I(X : Y ).Alternatively, we could take I to be the zero-error information.Yet, other possibilities include the Wyner common information [9] or the quantum mutual information [10].Generally, though, we require that I (X : Y ) = 0 if Y is a constant, which is satisfied by both the zero-error and Shannon information.
For a given choice of I, we seek a function that captures the amount of information about Y that is captured by each of the predictors X 1 , . . ., X n .We say that I ∩ is an intersection information for I if I ∩ (X : Y ) = I (X : Y ).There are currently 11 intuitive properties that we wish the ideal intersection information measure, I ∩ , to satisfy.Some are new (e.g., lower bound (LB), strong monotonicity (M 1 ), and equivalence-class invariance (Eq)), but most were introduced earlier, in various forms, in [1][2][3][4][5].They are as follows: • (Eq) Equivalence-class invariance: I ∩ (X 1 , . . ., X n : Y ) is invariant under substitution of X i (for any i = 1, . . ., n) or Y by an informationally equivalent random variable.
The next set of properties relate the intersection information to the chosen measure of information between X and Y .
• (LP 0 ) Weak local positivity: For n = 2 predictors, the derived "partial information" defined in [1] and described in Section 5 are nonnegative.If both (GP) and (M 0 ) are satisfied, as well as Finally, we have the "strong" properties: Strong local positivity: For all n, the derived "partial information" defined in [1] is nonnegative.
Properties (LB), (M 1 ) and (Eq) are introduced for the first time here.However, (Eq) is satisfied by most information-theoretic quantities and is implicitly assumed by others.Though absent from our list, it is worthwhile to also consider continuity and chain rule properties, in analogy with the mutual information [4,11].

Zero-Error Information
Introduced in [7], the zero-error information, or Gács-Körner common information, is a stricter variant of Shannon mutual information.Whereas the mutual information, I(A : B), quantifies the magnitude of information A conveys about B with an arbitrarily small error > 0, the zero-error information, denoted I 0 (A : B), quantifies the magnitude of information A conveys about B with exactly zero error, i.e., = 0.The zero-error information between A and B equals the entropy of the common random variable A B, Zero-error information has several notable properties, but the most salient is that it is nonnegative and bounded by the mutual information, 0 ≤ I 0 (A : B) ≤ I(A : B) .

Intersection Information for Zero-Error Information
For the zero-error information case (i.e., I = I 0 ), we propose the zero-error intersection information I 0 (X 1 , . . ., X n : Y ) as the maximum zero-error information, I 0 (Q : Y ), that a random variable, Q, conveys about Y , subject to Q being a function of each predictor X 1 , . . ., X n : ( In Lemma 7 of Appendix 7.3, it is shown that the common random variable across all predictors is the maximizing Q.This simplifies Equation (2) to: Most importantly, the zero-error information I 0 (X 1 , . . ., X n : Y ) satisfies nine of the 11 desired properties from Section 2.3, leaving only (LP 0 ) and (LP 1 ) unsatisfied.See Lemmas 1, 2, and 3 in Appendix 7.3 for details.

Candidate Intersection Information for Shannon Information
In the last section, we defined an intersection information for zero-error information that satisfies the vast majority of the desired properties.This is a solid start, but an intersection information for Shannon mutual information remains the goal.Towards this end, we use the same method as in Equation ( 2), leading to I , our candidate intersection information measure for Shannon mutual information: In Lemma 8 of Appendix 7.3, it is shown that Equation (4) simplifies to: Unfortunately, I does not satisfy as many of the desired properties as I 0 .However, our candidate, I , still satisfies seven of the 11 properties; most importantly, the coveted (TM) that, until now, had not been satisfied by any proposed measure.See Lemmas 4, 5 and 6 in Appendix 7.3 for details.Table 1 lists the desired properties satisfied by I min , I and I 0 .
For reference, we also include I red , the proposed measure from [3].
Despite not satisfying (LP 0 ), I remains an important stepping-stone towards the ideal Shannon I ∩ .First, I captures what is inarguably redundant information (the common random variable); this makes I necessarily a lower bound on any reasonable redundancy measure.Second, it is the first proposal to satisfy target monotonicity.Lastly, I is the first measure to reach intuitive answers in many canonical situations, while also being generalizable to an arbitrary number of inputs.

Three Examples Comparing I min and I
Example Unq illustrates how I min gives undesirable (some claim fatally so [2]) decompositions of redundant and synergistic information.Examples Unq and RdnXor illustrate I 's successes and example ImperfectRdn illustrates I 's paramount deficiency.For each, we give the joint distribution Pr(x 1 , x 2 , y), a diagram and the decomposition derived from setting I min or I as the I ∩ measure.At each lattice junction, the left number is the I ∩ value of that node, and the number in parentheses is the I ∂ value (this is the same notation used in [4]).Readers unfamiliar with the n = 2 partial information lattice should consult [1], but in short, I ∂ measures the magnitude of "new" information at this node in the lattice beyond the nodes lower in the lattice.Specifically, the mutual information between the pair, X 1 X 2 and Y , decomposes into four terms: In order, the terms are given by the redundant information that X 1 and X 2 both provide to Y , the unique information that X 1 provides to Y , the unique information that X 2 provides to Y and finally, the synergistic information that X 1 and X 2 jointly convey about Y .Each of these quantities can be written in terms of standard mutual information and the intersection information, I ∩ , as follows: These quantities occupy the bottom, left, right and top nodes in the lattice diagrams, respectively.Except for ImperfectRdn, measures I and I 0 reach the same decomposition for all presented examples.

Example Unq (Figure 1)
The desired decomposition for example Unq is two bits of unique information; X 1 uniquely specifies one bit of Y , and X 2 uniquely specifies the other bit of Y .The chief criticism of I min in [2] was that I min calculated one bit of redundancy and one bit of synergy for Unq (Figure 1c).We see that unlike I min , I satisfyingly arrives at two bits of unique information.This is easily seen by the inequality, Therefore, as I(X 1 : X 2 ) = 0, we have I (X 1 , X 2 : Y ) = 0 bits leading to I ∂ (X 1 : Y ) = 1 bit and I ∂ (X 2 : Y ) = 1 bit (Figure 1d).

Example RdnXor (Figure 2)
In [2], RdnXor was an example where I min shined by reaching the desired decomposition of one bit of redundancy and one bit of synergy.We see that I finds this same answer.I extracts the common random variable within X 1 and X 2 -the r/R bit-and calculates the mutual information between the common random variable and Y to arrive at I (X 1 , X 2 : Y ) = 1 bit.

Example ImperfectRdn (Figure 3)
ImperfectRdn highlights the foremost shortcoming of I : It does not detect "imperfect" or "lossy" correlations between X 1 and X 2 .Given (LP 0 ), we can determine the desired decomposition analytically.First, I(X 1 X 2 : Y ) = I(X 1 : Y ) = 1 bit, and thus, I X 2 : Y |X 1 = 0 bits.Since the conditional mutual information is the sum of the synergy I ∂ (X 1 , X 2 : Y ) and unique information I ∂ (X 2 : Y ), both quantities must also be zero.Then, the redundant information Having determined three of the partial informations, we compute the final unique information: 1 /8 r1 r0 r1 1 /8 r1 r1 r0 1 /8 1 ( 1) .99 (.99) 1 (.01) .99(0) .99 (.99) How well do I min and I match the desired decomposition of ImperfectRdn?We see that I min calculates the desired decomposition (Figure 3c); however, I does not (Figure 3d).Instead, I calculates zero redundant information, that I ∩ (X 1 , X 2 : Y ) = 0 bits.This unpleasant answer arises from Pr(X 1 = 0, X 2 = 1) > 0. If this were zero, then both I and I min reach the desired one bit of redundant information.Due to the nature of the common random variable, I only sees the "deterministic" correlations between X 1 and X 2 ; add even an iota of noise between X 1 and X 2 , and I plummets to zero.This highlights the fact that I is not continuous: an arbitrarily small change in the probability distribution can result in a discontinuous jump in the value of I .As with traditional information measures, such as the entropy and the mutual information, it may be desirable to have an I ∩ measure that is continuous over the simplex.
To summarize, ImperfectRdn shows that when there are additional "imperfect" correlations between A and B, i.e., I(A : B|A B) > 0, I sometimes underestimates the ideal I ∩ (A, B : Y ).

Negative Synergy
In ImperfectRdn, we saw I calculate a synergy of −0.99 bits (Figure 3d).What does this mean?Could negative synergy be a "real" property of Shannon information?When n = 2, it is fairly easy to diagnose the cause of negative synergy from the equation for I ∂ (X 1 X 2 : Y ) in Equation (7).Given (GP), negative synergy occurs if and only if, where I ∪ is dual to I ∩ and related by the inclusion-exclusion principle.For arbitrary n, this is I ∪ (X 1 , . . ., X n : Y ) ≡ S⊆{X1,...,Xn} (−1) |S|+1 I ∩ S 1 , . . ., S |S| : Y .The intuition behind I ∪ is that it represents the aggregate information contributed by the sources, X 1 , . . ., X n , without considering synergies or double-counting redundancies.
From Equation ( 9), we see that negative synergy occurs when I ∩ is small, probably too small.Equivalently, negative synergy occurs when the joint random variable conveys less about Y than the sources, X 1 and X 2 , convey separately; mathematically, when On the face of it, this sounds strange.No structure "disappears" after X 1 and X 2 are combined by the operator.By the definition of , there are always functions f 1 and f 2 , such that X 1 ∼ = f 1 (Z) and X 2 ∼ = f 2 (Z).Therefore, if your favorite I ∩ measure does not satisfy (LP 0 ), it is too strict.
This means that our measure, I 0 , does not account for the full zero-information overlap between I 0 (X 1 : Y ) and I 0 (X 2 : Y ).This is shown in the example, Subtle (Figure 4), where I 0 calculates a synergy of −0.252 bits.Defining a zero-error, I ∩ , that satisfies (LP 0 ) is a matter of ongoing research.

Conclusions and Path Forward
We made incremental progress on several fronts towards the ideal Shannon I ∩ .

Desired Properties
We have expanded, tightened and grounded the desired properties for I ∩ .Particularly, • (LB) highlights an uncontentious, yet tighter lower bound on I ∩ than (GP).
• What was before an implicit assumption, we introduced (Eq) to better ground one's thinking.

A New Measure
Based on the Gács-Körner common random variable, we introduced a new Shannon I ∩ measure.Our measure, I , is theoretically principled and the first to satisfy (TM).A point to keep in mind is that our intersection information is zero whenever the distribution Pr(x 1 , x 2 , y) has full support; this dependence on structural zeros is inherited from the common random variable.

How to Improve
We identified where I fails; it does not detect "imperfect" correlations between X 1 and X 2 .One next step is to develop a less stringent I ∩ measure that satisfies (LP 0 ) for ImperfectRdn, while still satisfying (TM).Satisfying continuity would also be a good next step.
Contrary to our initial expectation, Subtle, showed that I 0 does not satisfy (LP 0 ).This matches a result from [4], which shows that (LP 0 ), (S 1 ), (M 0 ) and (Id) cannot all be simultaneously satisfied, and it suggests that I 0 is too strict.

Appendix
By and large, most of these proofs follow directly from the lattice properties and also from the invariance and monotonicity properties with respect to ∼ = and .
Proof.(GP) follows immediately from the nonnegativity of the entropy.(Eq) follows from the invariance of entropy within the equivalence classes induced by ∼ =. (TM) follows from the monotonicity of the entropy with respect to .(M 0 ) also follows from the monotonicity of the entropy, but now applied to i X i W Y i X i Y .If there exists some j, such that X j W , then generalized absorption says that i X i W Y ∼ = i X i Y , and thus, we have the equality condition.(S 0 ) is a consequence of the commutativity of the operator.To see that (LP 0 ) is not satisfied by the I 0 , we point to the example, Subtle (Figure 4), which has negative synergy.One can also rewrite (LP 0 ) as the supermodularity law for common information, which is known to be false in general.(See [8], Section 5.4.)Lemma 2. I 0 (X 1 , . . ., X n : Y ) satisfies (LB), (SR), and (Id).
Proof.For (LB), note that Q X 1 • • • X n for any Q obeying Q X i for i = 1, . . ., n.Then, apply the monotonicity of the entropy.(SR) is trivially true given Lemma 7 and the definition of zero-error information.Finally, (Id) follows from the absorption law and the invariance of the entropy.Lemma 3. I 0 (X 1 , . . ., X n : Y ) satisfies (M 1 ) and (S 1 ), but not (LP 1 ).
Proof.(M 1 ) follows using the absorption and monotonicity of the entropy in nearly the same way that (M 0 ) does.(S 1 ) follows from commutativity, and (LP 1 ) is false, because (LP 0 ) is false.
Proof.(GP) follows from the nonnegativity of mutual information.(Eq) follows from the invariance of entropy.(TM) follows from the data processing inequality.(M 0 ) follows from applying the monotonicity of the mutual information I(Y : • ) to i X i W i X i .If there exists some j, such that X j W , then generalized absorption says that i X i W ∼ = i X i , and thus, we have the equality condition.(S 0 ) follows from commutativity, and a counterexample for (LP 0 ) is given by ImperfectRdn (Figure 3).Lemma 5. I (X 1 , . . ., X n : Y ) satisfies (LB) and (SR), but not (Id).

Figure 1 :
Figure 1: Example Unq.This is the canonical example of unique information.X 1 and X 2 each uniquely specify a single bit of Y .This is the simplest example, where I min calculates an undesirable decomposition (c) of one bit of redundancy and one bit of synergy.I and I 0 each calculate the desired decomposition (d).(a) Distribution and information quantities; (b) circuit diagram; (c) I min ; (d) I and I 0 .

Figure 2 :
Figure 2: Example RdnXor.This is the canonical example of redundancy and synergy coexisting.I min and I each reach the desired decomposition of one bit of redundancy and one bit of synergy.This is the simplest example demonstrating I and I 0 correctly extracting the embedded redundant bit within X 1 and X 2 .(a) Distribution and information quantities; (b) circuit diagram; (c) I min ; (d) I and I 0 .

Figure 3 :
Figure 3: Example ImperfectRdn.I is blind to the noisy correlation between X 1 and X 2 and calculates zero redundant information.An ideal I ∩ measure would detect that all of the information X 2 specifies about Y is also specified by X 1 to calculate I ∩ (X 1 , X 2 : Y ) = 0.99 bits.(a) Distribution and information quantities; (b) circuit diagram; (c) I min ; (d) I ; (e) I 0 .

Figure 4 :
Figure 4: Example Subtle.In this example, both I and I 0 calculate a synergy of −0.252 bits of synergy.What kind of redundancy must be captured for a nonnegative decomposition for this example?(a) Distribution and information quantities; (b) circuit diagram; (c) I min ; (d) I and I 0 .

Table 1 :
The I ∩ desired properties that each measure satisfies.(The appendices provide proofs for I and I 0 .) Therefore, what kind of zero-error informational overlap is I 0 not capturing?The answer is of paramount importance.The next step is to formalize what exactly is required for a zero-error I ∩ to satisfy (LP 0 ).From Subtle, we can likewise see that within zero-error information, (Id) and (LP 0 ) are incompatible.wasfunded by a Department of Energy Computational Science Graduate Fellowship; Edwin K. P. Chong was funded by Colorado State University's Information Science & Technology Center; Ryan G. James and James P. Crutchfield were funded by Army Research Office grant W911NF-12-1-0234; Christopher J. Ellison was funded by a subaward from the Santa Fe Institute under a grant from the John Templeton Foundation.