Decomposing and Tracing Mutual Information by Quantifying Reachable Decision Regions

Abstract

The idea of a partial information decomposition (PID) gained significant attention for attributing the components of mutual information from multiple variables about a target to being unique, redundant/shared or synergetic. Since the original measure for this analysis was criticized, several alternatives have been proposed but have failed to satisfy the desired axioms, an inclusion–exclusion principle or have resulted in negative partial information components. For constructing a measure, we interpret the achievable type I/II error pairs for predicting each state of a target variable (reachable decision regions) as notions of pointwise uncertainty. For this representation of uncertainty, we construct a distributive lattice with mutual information as consistent valuation and obtain an algebra for the constructed measure. The resulting definition satisfies the original axioms, an inclusion–exclusion principle and provides a non-negative decomposition for an arbitrary number of variables. We demonstrate practical applications of this approach by tracing the flow of information through Markov chains. This can be used to model and analyze the flow of information in communication networks or data processing systems.

1. Introduction

A Partial Information Decomposition (PID) aims to attribute the provided information about a discrete target variable T from a set of predictor or viewable variables $\mathbf{V}=\{V_1,\dots ,V_n\}$ to each individual variable $V_i$. The partial contributions to the information about T may be provided by all variables (redundant or shared), by a specific variable (unique) or only be available through a combination of variables (synergetic/complementing) [1]. This decomposition is particularly applicable when studying complex systems. For example, it can be used to study logical circuits, neural networks [2] or the propagation of information over multiple paths through a network. The concept of synergy has been applied to develop data privacy techniques [3,4], and we think that the concept of redundancy may be suitable to study a notion of robustness in data processing systems.

Unfortunately and to the best of our knowledge, there does not exist a non-negative decomposition of mutual information for an arbitrary number of variables that satisfies the commutativity, monotonicity and self-redundancy axioms except the original measure of Williams and Beer [5]. However, this measure has been criticized for not distinguishing “the same information and the same amount of information” [6,7,8,9].

Here, we propose an alternative non-negative partial information decomposition that satisfies Williams and Beer’s axioms [5] for an arbitrary number of variables. It provides an intuitive operational interpretation and results in an algebra like probability theory. To demonstrate that the approach distinguishes the same information from the same amount of information, we highlight its application in tracing the flow of information through a Markov chain, as visualized in Figure 1.

This work is structured in three parts: Section 2 provides an overview of the related work and background information. Section 3 presents a representation of pointwise uncertainty, constructs a distributive lattice and demonstrates that mutual information is the expected value of its consistent valuation. Section 4 discusses applications of the resulting measure to PIDs and the tracing of information through Markov chains. We provide an overview of the used notation at the end of the paper.

2. Related Work

We briefly summarize partial orders and the four main publications which led to our proposed decomposition approach. This includes the PID by Williams and Beer [5], the quantification of unique information by Bertschinger et al. [10] and Griffith and Koch [11], the Blackwell order based on Bertschinger and Rauh [12], the evaluation of binary decision problems using Receiver Operating Characteristics and consistent lattice valuations by Knuth [13].

2.1. Partial Orders and Lattices

This section provides a brief overview of the relevant definitions on partial orders and lattices for the context of this work based on [9,13]. A binary ordering relation ≼ on a set $\mathbf{L}$ is called a preorder if it is reflexive and transitive. If the ordering relation additionally satisfies an antisymmetry, then $(\mathbf{L},\preccurlyeq )$ is called a partially ordered set (poset). For $\alpha ,\beta ,\gamma \in \mathbf{L}$:

$\alpha \preccurlyeq \alpha$	(reflexivity)
$\mathrm{if}\alpha \preccurlyeq \beta \mathrm{and}\beta \preccurlyeq \gamma \mathrm{then}\alpha \preccurlyeq \gamma$	(transitivity)
$\mathrm{if}\alpha \preccurlyeq \beta \mathrm{and}\beta \preccurlyeq \alpha \mathrm{then}\alpha =\beta$	(antisymmetry)

Two elements satisfy $\alpha \preccurlyeq \beta$, $\beta \preccurlyeq \alpha$ or may be incomparable, meaning $\alpha ⋠\beta$ and $\beta ⋠\alpha$. A partially ordered set has a bottom element $\perp \in \mathbf{L}$ if $\perp \preccurlyeq \alpha$ for all $\alpha \in \mathbf{L}$ and a top element $\top \in \mathbf{L}$ if $\alpha \preccurlyeq \top$ for all $\alpha \in \mathbf{L}$. For each element $\alpha$, it can be defined a down-set ($\downarrow \alpha$) and up-set ($\uparrow \alpha$) as well as a strict down-set ($\dot{\downarrow }\alpha$) and strict up-set ($\dot{\uparrow }\alpha$) as shown below:

$$\begin{array}{ccc}\hfill \downarrow \alpha & =\{\beta \in \mathbf{L}\mid \beta \preccurlyeq \alpha \}\hfill & \hfill \left(\mathrm{down}\text{-}\mathrm{set}\right)\\ \hfill \dot{\downarrow }\alpha & =\{\beta \in \mathbf{L}\mid \beta \preccurlyeq \alpha \mathrm{and}\alpha ⋠\beta \}\hfill & \hfill \left(\mathrm{strict}\mathrm{down}\text{-}\mathrm{set}\right)\\ \hfill \uparrow \alpha & =\{\beta \in \mathbf{L}\mid \alpha \preccurlyeq \beta \}\hfill & \hfill \left(\mathrm{up}\text{-}\mathrm{set}\right)\\ \hfill \dot{\uparrow }\alpha & =\{\beta \in \mathbf{L}\mid \alpha \preccurlyeq \beta \mathrm{and}\beta ⋠\alpha \}\hfill & \hfill \left(\mathrm{strict}\mathrm{up}\text{-}\mathrm{set}\right)\end{array}$$

A lattice is a partially ordered set $(\mathbf{L},\preccurlyeq )$ for which every pair of elements $\{\alpha ,b\}\subseteq \mathbf{L}$ has a unique least upper bound $\alpha ⋎\beta =sup\{\alpha ,\beta \}$, referred to as joint, and a unique greatest lower bound $\alpha ⋏\beta =inf\{\alpha ,\beta \}$, referred to as meet. This creates an algebra $(\mathbf{L},⋏,⋎)$ with the binary operators ⋏ and ⋎ that satisfies indempotency, commutativity, associativity and absorption. The consistency relates the ordering relation and algebra with each other. A distributive lattice additionally satisfies distributivity.

$\alpha ⋎\alpha =\alpha$	$\alpha ⋏\alpha =\alpha$	(indempotency)
$\alpha ⋎\beta =\beta ⋎\alpha$	$\alpha ⋏\beta =\beta ⋏\alpha$	(commutativity)
$\alpha ⋎(\beta ⋎\gamma )=(\alpha ⋎\beta )⋎\gamma$	$\alpha ⋏(\beta ⋏\gamma )=(\alpha ⋏\beta )⋏\gamma$	(associativity)
$\alpha ⋎(\alpha ⋏\beta )=\alpha$	$\alpha ⋏(\alpha ⋎\beta )=\alpha$	(absorption)
$\alpha \preccurlyeq \beta \Rightarrow \alpha ⋎\beta =\beta$	$\alpha \preccurlyeq \beta \Rightarrow \alpha ⋏\beta =\alpha$	(consistency)
$\alpha ⋎(\beta ⋏\gamma )=(\alpha ⋎\beta )⋏(\alpha ⋎\gamma )\hspace{1em}$	$\alpha ⋏(\beta ⋎\gamma )=(\alpha ⋏\beta )⋎(\alpha ⋏\gamma )$	(distributivity)

2.2. Partial Information Decomposition

This section summarizes Williams and Beer’s general approach to partial information decompositions [5]. A more detailed discussion of the literature and required background can be found in [9] (pp. 6–20).

Williams and Beer [5] define sources ${\mathbf{S}}_i\in {\mathcal{P}}_1\left(\mathbf{V}\right)$ as all combinations of viewable variables (${\mathcal{P}}_1\left(\mathbf{V}\right)$ referring to the power set of $\mathbf{V}$ without the empty set) and use Equation (1a) to construct all distinct interactions between them $\alpha \in \mathcal{A}\left(\mathbf{V}\right)$, which are referred to as partial information atoms. Equation (1b) provides a partial order of atoms to construct a redundancy lattice ($\mathcal{A}\left(\mathbf{V}\right),\preccurlyeq$). As a convention, we indicate the visible variables contained in a source by its index, such as ${\mathbf{S}}_{12}=\{V_1,V_2\}$. The example of the redundancy lattice for two and three visible variables is shown in Figure 2.

$$\begin{array}{c}\hfill \mathcal{A}\left(\mathbf{V}\right)=\{\alpha \in {\mathcal{P}}_1\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right)\mid \forall {\mathbf{S}}_i,{\mathbf{S}}_j\in \alpha ,{\mathbf{S}}_i\not\subset {\mathbf{S}}_j\},\end{array}$$

(1a)

$$\begin{array}{c}\hfill \forall \alpha ,\beta \in \mathcal{A}\left(\mathbf{V}\right),(\alpha \preccurlyeq \beta \iff \forall {\mathbf{S}}_j\in \beta ,\exists {\mathbf{S}}_i\in \alpha \mid {\mathbf{S}}_i\subseteq {\mathbf{S}}_j).\end{array}$$

(1b)

A measure of redundant information $I_{\cap }$ shall be defined for this lattice as “[…] cumulative information function which in effect integrates the contribution from each node as one moves up through the nodes of the lattice” [9] (p. 15). Williams and Beer [5] then use the Möbius inverse (Equation (2)) to identify the partial information $I_{\delta }(\alpha ;T)$ as the contribution of atom $\alpha \in \mathcal{A}\left(\mathbf{V}\right)$ and therefore the desired unique/redundant/synergetic component. A PID is said to be non-negative if the resulting partial contributions are guaranteed to be non-negative.

$$I_{\delta }(\alpha ;T)=I_{\cap }(\alpha ;T)-\sum _{\beta \in \dot{\downarrow }\alpha }{I}_{\delta }(\beta ;T).$$

(2)

Williams and Beer [5] highlight three axioms that a measure of redundancy should satisfy.

Axiom 1

(Commutativity). Invariant to the order of sources (σ permuting the order of indices):

$$I_{\cap }({\mathbf{S}}_1,\dots ,{\mathbf{S}}_i;T)=I_{\cap }({\mathbf{S}}_{\sigma \left(1\right)},\dots ,{\mathbf{S}}_{\sigma \left(i\right)};T)$$

Axiom 2

(Monotonicity). Additional sources can only decrease redundant information:

$$I_{\cap }({\mathbf{S}}_1,\dots ,{\mathbf{S}}_{i-1};T)\ge I_{\cap }({\mathbf{S}}_1,\dots ,{\mathbf{S}}_i;T)$$

Axiom 3

(Self-redundancy). For a single source, redundancy equals mutual information:

$$I_{\cap }({\mathbf{S}}_i;T)=I({\mathbf{S}}_i;T)$$

Finally, Williams and Beer [5] proposed $I_{\min }$ (Equation (3)) as a measure of redundancy and demonstrated that it satisfies the required axioms.

$$\begin{array}{cc}\hfill I({\mathbf{S}}_i;T=t)& =\sum _{s\in {\mathcal{S}}_i}p(s\mid t)\left[log\frac{1}{p\left(t\right)}-log\frac{1}{p(t\mid s)}\right]\hfill \end{array}$$

(3a)

$$\begin{array}{cc}\hfill I_{\min }({\mathbf{S}}_1,\dots ,{\mathbf{S}}_k;T)& =\sum _{t\in \mathcal{T}}p\left(t\right)\underset{i\in 1..k}{min}I({\mathbf{S}}_i;T=t).\hfill \end{array}$$

(3b)

However, the measure has been criticized for not distinguishing “the same information and the same amount of information” [6,7,8,9] due to its use of a pointwise minimum (for each $t\in \mathcal{T}$) over the sources.

2.3. Quantifying Unique Information

A non-negative decomposition for the case of two viewable variables $\mathbf{V}=\{V_1,V_2\}$ was proposed by Bertschinger et al. [10] (defining unique information) as well as an equivalent decomposition by Griffith and Koch [11] (defining union information) as shown in Equation (4) (modified notation). The function $\vartheta (V_1,V_2;T)$ acts as an information measure of the union for $V_1$ and $V_2$ (the minimal information that any two variables with the same marginal distributions can achieve), which is then used to compute the partial contributions using an inclusion–exclusion principle. Bertschinger et al. [10] motivated the decomposition from the operational interpretation that if a variable provides unique information, there must be a way to utilize this information in a decision problem for some reward function. Additionally, they argue that unique information should only depend on the marginal distributions $P_{(T,V_1)}$ and $P_{(T,V_2)}$.

$$\begin{array}{cc}\hfill \vartheta (V_1,V_2;T)=& minI(F;G_1,G_2)\mathrm{s}.\mathrm{t}.P_{(F,G_1)}=P_{(T,V_1)}\mathrm{and}{P}_{(F;G_2)}=P_{(T,V_2)}\hfill \\ \hfill S(V_1,V_2;T)& =I(V_1;T)+I(V_2;T)-\vartheta (V_1,V_2;T)\hfill & \hfill \left(\mathrm{Shared}\right)\\ \hfill U(V_1;T)& =\vartheta (V_1,V_2;T)-I(V_1;T)\hfill & \hfill \left(\mathrm{Unique}\right)\\ \hfill C(V_1,V_2;T)& =I(T;V_1,V_2)-\vartheta (V_1,V_2;T)\hfill & \hfill \left(\mathrm{Complementing}\right)\end{array}$$

(4)

We highlight this decomposition since our approach can be interpreted as its pointwise extension (see Section 4.1).

2.4. Blackwell Order

A channel $\kappa$ can be represented as a (row) stochastic matrix wherein each element is non-negative and all rows sum to one (see Figure 3). In this work, we consider the sources to be the indirect observation of the target variable through a channel $T\stackrel{{\kappa }_i}{\to }{\mathbf{S}}_i$ while taking the joint distribution of the visible variables within ${\mathbf{S}}_i$. As a result, ${\kappa }_i$ is obtained from the conditional probability distribution ${\kappa }_i=T\to {\mathbf{S}}_i=P_{\left({\mathbf{S}}_i\right|T)}$. As for sources, we list the contained visible variables as an index such that $i=12$ corresponds to ${\kappa }_{12}=P_{\left({\mathbf{S}}_{12}\right|T)}=P_{(V_1,V_2|T)}$.

The Blackwell order ${\kappa }_1⊑{\kappa }_2$ is a preorder of channels, as shown in Equation (5) [14]. It highlights that a channel equivalent to ${\kappa }_1$ can be obtained by garbling the output of ${\kappa }_2$ (a chaining of channels as seen in Equation (5)). Therefore, there exists a decision strategy based on ${\kappa }_2$ for any reward function that performs at least as well as all strategies based on ${\kappa }_1$ [12].

$${\kappa }_1⊑{\kappa }_2⟺{\kappa }_1={\kappa }_2·\lambda \mathrm{for}\mathrm{some}\mathrm{channel}\lambda$$

(5)

Bertschinger and Rauh [12] showed that the Blackwell order does not define a lattice in general since it does not provide a unique meet and joint element beyond binary inputs. However, binary input channels provide a special case for which the Blackwell order is equivalent to the zonotope order and defines a lattice. We use the notation ${\kappa }^t$ to indicate that a channel has a binary input ($\left|\mathcal{T}\right|=2$) or corresponds to the one-vs-rest encoding for one state t if $\left|\mathcal{T}\right|>2$. In this case, the row stochastic matrix representing a channel contains a set of vectors ${\overrightarrow{v}}_s$ as shown in Equation (6). A zonotope $Z_{{\kappa }^t}$ (Equation (6b)) corresponds to “the image of the unit cube […] under the linear map corresponding to [${\kappa }^t$]” [12] (p. 2), and the resulting zonotope order ${\kappa }_1^t⊑{\kappa }_2^t\iff Z_{{\kappa }_1^t}\subseteq Z_{{\kappa }_2^t}$ is a preorder that is identical to the Blackwell order in the special case of binary input channels [12] as visualized in Figure 3. In the resulting lattice, the joint of two channels can be obtained as the convex hull $Z_{{\kappa }_1^t\bigsqcup {\kappa }_2^t}$ of the zonotopes $Z_{{\kappa }_1^t}$ and $Z_{{\kappa }_2^t}$, and the meet element $Z_{{\kappa }_1^t\sqcap {\kappa }_2^t}$ corresponds to their intersection.

$$\begin{array}{c}\hfill {\kappa }_i^t=\left[\begin{array}{cccc}p({\mathbf{S}}_i=s_1\mid T=t)& p({\mathbf{S}}_i=s_2\mid T=t)& \dots & p({\mathbf{S}}_i=s_n\mid T=t)\\ p({\mathbf{S}}_i=s_1\mid T\ne t)& p({\mathbf{S}}_i=s_2\mid T\ne t)& \dots & p({\mathbf{S}}_i=s_n\mid T\ne t)\end{array}\right]\end{array}$$

(6a)

$$\begin{array}{c}\hfill Z_{{\kappa }_i^t}=\left\{\sum _{s\in {\mathcal{S}}_i}{x}_s·{\overrightarrow{v}}_s\mid 0\le x_s\le 1\right\}\mathrm{where}{\overrightarrow{v}}_s=\left(\begin{array}{c}p({\mathbf{S}}_i=s\mid T=t)\\ p({\mathbf{S}}_i=s\mid T\ne t)\end{array}\right)\end{array}$$

(6b)

2.5. Receiver Operating Characteristic Curves

While any classification system can be represented as channel, this section focuses on binary decision problems or the one-vs-rest encoding of others ($T^t=t\iff T=t$). The binary label $t\in {\mathcal{T}}^t$ is used to obtain a sample $s\in {\mathcal{S}}_i$, which is processed by a classification system C to its output $o\in \mathcal{O}$ with $o=C\left(s\right)$, and applying a decision strategy d shall result in an approximation of the label $\widehat{t}\in {\mathcal{T}}^t$ with $\widehat{t}=d\left(o\right)$. This forms the Markov chain: $T^t\to {\mathbf{S}}_i\to O\to {\widehat{T}}^t$. A common method of analyzing binary decision/classification systems is the Receiver Operating Characteristic (ROC). A ROC plot typically represents a classifier C with a continuous, discrete or categorical output range (by assigning distinct arbitrary values to each category) for a binary decision problem by a curve in a True-Positive Rate (TPR)/False-Positive Rate (FPR) diagram for varying decision thresholds ${\tau }_x$ with the decision rule for a sample s being $C\left(s\right)\le {\tau }_x\iff \mathrm{False}$ [15]. The resulting points are typically connected using a step function, as shown in red in Figure 4a. As a result of using a single decision threshold, the points of the ROC curve monotonically increase from $(0,0)$ to $(1,1)$; however, they are in general neither concave nor convex [16].

We want to highlight the distinction between a ROC curve and the achievable performance pairs (TPR, FPR) based on the classifier. Any performance pair within the convex hull of the obtained points for constructing the ROC curve can be achieved since the decision strategy of Equation (7) results in an interpolation of the points corresponding to ${\tau }_1\le {\tau }_2$ with the parameter $0\le h\le 1$ in the TPR/FPR diagram. Therefore, while a ROC curve is not convex in general, the achievable performance region is convex in general.

$$\begin{array}{cc}\hfill C\left(s\right)\le {\tau }_1& \Rightarrow \mathrm{False},\hfill \\ \hfill {\tau }_1<C\left(s\right)\le {\tau }_2& \Rightarrow \mathrm{Bernoulli}\left(h\right),\hfill \\ \hfill {\tau }_2<C\left(s\right)& \Rightarrow \mathrm{True}.\hfill \end{array}$$

(7)

When utilizing the set of all available thresholds on the classification output, we can identify the reachable decision regions within the TPR/FPR diagram using the likelihood ratio test, which is well known to be optimal for binary decision problems: Neyman–Pearson theory [17] states that the likelihood ratio test (Equation (8)) provides the minimal type II error (minimal $\mathit{\beta }$, maximal TPR $=\overline{\mathit{\beta }}=1-\mathit{\beta }$) for a bounded type I error (FPR, $\mathit{\alpha }$).

$$\begin{array}{cc}\hfill \frac{P({\mathbf{S}}_i=s|T=t)}{P({\mathbf{S}}_i=s|T\ne t)}<\tau & \Rightarrow \mathrm{False},\hfill \\ \hfill \frac{P({\mathbf{S}}_i=s|T=t)}{P({\mathbf{S}}_i=s|T\ne t)}=\tau & \Rightarrow \mathrm{Bernoulli}\left(h\right),\hfill \\ \hfill \frac{P({\mathbf{S}}_i=s|T=t)}{P({\mathbf{S}}_i=s|T\ne t)}>\tau & \Rightarrow \mathrm{True}.\hfill \end{array}$$

(8)

Notice that the decision criterion is determined by the slope of each vector in the row stochastic matrix that represents the binary input channel (Equation (6a)). This effective reordering of vectors based on their slope when varying the parameters $\tau$ and h results in the upper half of the zonotope discussed in Section 2.4 and as visualized in Figure 4a. The lower half of the zonotope is obtained from negating the outcome of the likelihood ratio test. Therefore, the zonotope representation of a channel corresponds to the achievable performance region in a TPR/FPR diagram of a classifier at binary decision problems. When reconsidering Figure 3, the channels ${\kappa }_1^t=T^t\to O_1$ and ${\kappa }_2^t=T^t\to O_2$ may correspond to two classifiers $C_1$ and $C_2$ whose channel parameters have been estimated from a ROC curve, and the achievable performance regions correspond to the zonotopes in a TPR/FPR diagram. Since the likelihood ratio test is optimal for binary decision problems, there cannot exist a decision strategy that would achieve a performance outside the zonotope. At the same time, the likelihood ratio test can be randomized to reach any desired position within the zonotope.

Finally, notice that the convex hull of any two classification systems is reachable by their randomized combination. We can view each classifier as an observation from a channel ${\kappa }_1^t$/${\kappa }_2^t$ about $T^t$ and know that there always exists a garbling $\lambda$ of the joint channel ${\kappa }_{12}^t$ to obtain their convex hull ${\kappa }_1^t\bigsqcup {\kappa }_2^t={\kappa }_{12}^t\lambda$. Using a likelihood ratio test on ${\kappa }_1^t\bigsqcup {\kappa }_2^t$, any position within the convex hull is reachable as a randomized combination of both classifiers. This has been visualized in Figure 4b. Due to this reason, we will say in Section 3.1 that the convex hull should be fully attributed to the marginal channels ${\kappa }_1^t$ and ${\kappa }_2^t$.

2.6. Lattice Valuations

This section summarizes the properties of consistent lattice valuations based on Knuth [13]. The quantification of a lattice $(\mathbf{L},\preccurlyeq )$ or $(\mathbf{L},⋏,⋎)$ with $\alpha ⋏\beta =\alpha \iff \alpha \preccurlyeq \beta$ for elements of the set $\alpha ,\beta \in \mathbf{L}$ is a function $q:\mathbf{L}\to \mathbb{R}$, which assigns reals to each element. A quantification is called a valuation if any two elements maintain an ordering relation: $\alpha \preccurlyeq \beta$ implies that $q\left(\alpha \right)\le q\left(\beta \right)$. A quantification q is consistent if it satisfies a sum rule (inclusion–exclusion principle): $q(\alpha ⋎\beta )=q\left(\alpha \right)+q\left(\beta \right)-q(a⋏\beta )$. If the bottom element of the lattice (⊥) is evaluated to zero $q(\perp )=0$, then the valuation of the Cartesian product of two lattices $q\left(\right(\alpha ;\beta \left)\right)=q\left(\alpha \right)·q\left(\beta \right)$ remains consistent with the individual lattices. Finally, a bi-quantification can be defined as $b(\alpha ,\beta )=q(\alpha ⋏\beta )/q\left(\beta \right)$. Similar to Knuth [13], we will use the notation $q\left(\right[\alpha ;\beta \left]\right)\equiv b(\alpha ,\beta )$ which can be thought of as quantifying a degree of inclusion for $\alpha$ within $\beta$. The distributive lattice then creates an algebra like probability theory for the consistent valuation, as summarized in Equation (9) [13].

$$\begin{array}{ccc}\hfill q(\alpha ⋎\beta )& =q\left(\alpha \right)+q\left(\beta \right)-q(\alpha ⋏\beta )\hfill & \hfill \left(\mathrm{Sum}\mathrm{rule}\right)\\ \hfill q\left(\right[\alpha ⋎\beta ;\gamma \left]\right)& =q\left(\right[\alpha ;\gamma \left]\right)+q\left(\right[\beta ;\gamma \left]\right)-q\left(\right[\alpha ⋏\beta ;\gamma \left]\right)\hfill & \hfill \left(\mathrm{Sum}\mathrm{rule}\right)\\ \hfill q\left(\right(\alpha ;\beta \left)\right)& =q\left(\alpha \right)·q\left(\beta \right)\hfill & \hfill \left(\mathrm{Direct}\mathrm{product}\mathrm{rule}\right)\\ \hfill q\left(\right([\alpha ;\beta ];[\tau ;\upsilon ]\left)\right)& =q\left(\right[\alpha ;\tau \left]\right)·q\left(\right[\beta ;\upsilon \left]\right)\hfill & \hfill \left(\mathrm{Direct}\mathrm{product}\mathrm{rule}\right)\\ \hfill q\left(\right[\beta ⋏\gamma ;\alpha \left]\right)& =q\left(\right[\gamma ;\alpha ⋏\beta \left]\right)·q\left(\right[\beta ;\alpha \left]\right)\hfill & \hfill \left(\mathrm{Product}\mathrm{rule}\right)\\ \hfill q\left(\right[\gamma ;\alpha ⋏\beta \left]\right)& =\frac{q\left(\right[\beta ;\alpha ⋏\gamma \left]\right)·q\left(\right[\gamma ;\alpha \left]\right)}{q\left(\right[\beta ;\alpha \left]\right)}\hfill & \hfill \left(\mathrm{Bayes}’ \mathrm{Theorem}\right)\end{array}$$

(9)

3. Quantifying Reachable Decision Regions

We start by studying the decomposition of binary decision problems from an interpretational perspective (Section 3.1). This provides the basis for constructing a distributive lattice in Section 3.2 and demonstrating the structure of a consistent valuation function. Section 3.3 highlights that mutual information is such a consistent valuation and extends the concept from binary decision problems to target variables with an arbitrary finite number of states. The resulting definition of shared information for the PID will be discussed as an application in Section 4.1 together with the tracing of information flows in Section 4.2.

We define an equivalence relation (∼) for binary input channels ${\kappa }^t$, which allows for the removal of zero vectors, the permutation of columns ($\mathbf{P}$ representing a permutation matrix) and the splitting/merging of columns with identical likelihood ratios (vectors of identical slope, $\ell \in \mathbb{R}$), as shown in Equation (10). These operations are invertible using garblings and do not affect the underlying zonotope.

$${\kappa }^t\sim \left[{\kappa }^t\left[\begin{array}{c}0\\ 0\end{array}\right]\right];$$

(10a)

$${\kappa }^t\sim {\kappa }^t\mathbf{P};$$

(10b)

$$[(1+\ell ){\overrightarrow{v}}_1{\overrightarrow{v}}_2\dots ]\sim \left[\begin{array}{cccc}{\overrightarrow{v}}_1& \ell {\overrightarrow{v}}_1& {\overrightarrow{v}}_2& \dots \end{array}\right].$$

(10c)

Based on this definition, block matrices cancel at an inverted sign ($\ell =-1$) if we allow negative columns, as shown in Equation (11), where ${\mathbf{M}}_1$ and ${\mathbf{M}}_2$ are some $2\times n$ matrix.

$${\mathbf{M}}_1\sim \left[\begin{array}{ccc}{\mathbf{M}}_1& {\mathbf{M}}_2& -{\mathbf{M}}_2\end{array}\right]$$

(11)

3.1. Motivation and Operational Interpretation

The aim of this section is to provide a first intuition based on a visual example for the methodology that will be used in Section 3.2 to construct a distributive lattice of the reachable decision regions and its consistent valuation. We only consider binary variables ${\mathcal{T}}^t=\{t,\overline{t}\}$ or the one-vs-rest encoding of others ($T^t=t\iff T=t$).

In the used example, the desired variable can be observed indirectly using the two variables $V_1$ and $V_2$. The visible variables are considered to be the output of the channels $T^t\stackrel{{\kappa }_1^t}{\to }{V}_1$, $T^t\stackrel{{\kappa }_2^t}{\to }{V}_2$ and $T^t\stackrel{{\kappa }_{12}^t}{\to }(V_1,V_2)$ and correspond to the zonotopes shown in Figure 5. We consider each reachable decision point (a pair of TPR and FPR) to represent a different notion of uncertainty about the state of the target variable. We want to attribute the reachable decision regions to each channel for constructing a lattice, as shown in Figure 6, with the following operational interpretation:

• Synergy: Corresponds to the partial contribution of ${\kappa }_{12}^t=T^t\to (V_1,V_2)$ and represents the decision region which is only accessible due to the (in-)dependence of both variables.
• Joint: The joint element ${\kappa }_1^t\vee {\kappa }_2^t=(T^t\to V_1)\vee (T^t\to V_2)$ corresponds to the joint under the Blackwell order and represents the decision region which is always accessible if the marginal distributions $(V_1,T^t)$ and $(V_2,T^t)$ can be obtained. Therefore, we say that its information shall be fully attributed to $V_1$ and $V_2$ such that is has no partial contribution. For binary target variables, this definition is equivalent to the notion of union information by Bertschinger et al. [10] and Griffith and Koch [11]. However, we extend the analysis beyond binary target variables with a different approach in Section 3.3.
• Unique: Corresponds to the partial contribution of ${\kappa }_1^t=T^t\to V_1$ or ${\kappa }_2^t=T^t\to V_2$ and represents the decision region that is lost when losing the variable. It only depends on their marginal distributions $(V_1,T^t)$ and $(V_2,T^t)$.
• Shared: Corresponds to the cumulative contribution of ${\kappa }_1^t\wedge {\kappa }_2^t=(T^t\to V_1)\wedge (T^t\to V_2)$ and represents the decision region which is lost when losing either $V_1$ or $V_2$. Since it only depends on the marginal distributions, we interpret it as being part of both variables. The shared decision region can be split in two components: the decision region that is part of both individual variables and the component that is part of the convex hull but neither individual one. The latter component only exists if both variables provide unique information.
• Redundant: The largest decision region ${\kappa }_1^t\sqcap {\kappa }_2^t=(T^t\to V_1)\sqcap (T^t\to V_2)$ which can be accessed from both $V_1$ and $V_2$. It corresponds to the meet under the Blackwell order and the part of shared information that can be represented by some random variable (pointwise extractable component of shared information). The redundant and shared regions are equal unless both variables provide some unique information.

Due to the invariance of re-ordering columns under the defined equivalence relation, ${\kappa }^t$ represents a set of likelihood vectors. All cumulative and partial decision regions of Figure 6 can be constructed using a convex hull operator (joint) and matrix concatenations under the defined equivalence relation (∼). For example, the shared decision region (meet) can be expressed through an inclusion–exclusion principle with the joint operator ${\kappa }_1^t\wedge {\kappa }_2^t\sim \left[\begin{array}{ccc}{\kappa }_1^t& {\kappa }_2^t& -{\kappa }_1^t\vee {\kappa }_2^t\end{array}\right]$. This operator is not closed on channels since it introduces negative likelihood vectors. Therefore, we distinguish the notation between channels (${\kappa }^t$) and atoms (${\alpha }^t$). These matrices ${\alpha }^t$ sum to one similar to channels but may contain negative columns. Their partial contributions ${\alpha }^{\delta t}$ sum to zero.

• The unique contribution of $V_2$:                          ${\alpha }^{\delta t}\sim \left[\begin{array}{cc}({\kappa }_1^t\vee {\kappa }_2^t)& -{\kappa }_1^t\end{array}\right]$
• The shared cumulative region of $V_1$ and $V_2$:   ${\beta }^t\sim \left[\begin{array}{ccc}{\kappa }_1^t& {\kappa }_2^t& -({\kappa }_1^t\vee {\kappa }_2^t)\end{array}\right]\sim {\kappa }_1^t\wedge {\kappa }_2^t$
• The shared partial contribution:                        ${\beta }^{\delta t}\sim \left[\begin{array}{cc}{\beta }^t& -({\kappa }_1^t\sqcap {\kappa }_2^t)\end{array}\right]$
• Each cumulative region corresponds to the combination of partial contributions in its down-set. Notice that the partial contribution of the shared region is canceled by a section of each unique contribution due to an opposing sign:

  $${\kappa }_1^t\sim \left[\begin{array}{ccc}{\alpha }^{\delta t}& {\beta }^{\delta t}& ({\kappa }_1^t\sqcap {\kappa }_2^t)\end{array}\right]$$

In Section 3.3, we demonstrate a valuation function f that can quantify all cumulative and partial atoms of this lattice while ensuring their non-negativity and consistency with the defined equivalence relation (∼). We will refer to a more detailed example on the valuation of partial decision regions in Appendix C in the context of the following section.

Why does the decomposition of reachable decision regions as shown in Figure 6 provide a meaningful operational interpretation? Because combining the partial contributions of the up-set for a variable results in the decision region that becomes inaccessible when the variable is lost, while combining the partial contributions of the down-set results in the decision region that is accessible through the variable. For example, losing access to variable $V_2$ results in losing access to the decision regions provided uniquely by $V_2$ and its synergy with $V_1$ (the up-set on the lattice). Additionally, the cumulative component corresponds to the combination of all partial contributions in its down-set since opposing vectors cancel under the defined equivalence relation (∼) such as the shared and unique contributions. Therefore, we define a consistent valuation of this lattice in Section 3.2 by quantifying decision regions based on their spanning vectors and highlight that the expected value for each $t\in \mathcal{T}$ corresponds to the definition of mutual information.

Section 3.2 and Section 3.3 focus only on defining the meet and joint operators ($\wedge /\vee$) with their consistent valuation. To obtain the pointwise redundant and synergetic components for a PID, we can later add the corresponding channels when constructing the pointwise lattices $\mathbf{V}=\{V_1,V_2,(V_1,V_2),V_1\sqcap V_2\}$ with the ordering of Figure 6 from the meet and joint operators.

3.2. Decomposition Lattice and Its Valuation

This section first defines the meet and joint operators (∧, ∨) and then constructs a consistent valuation for the resulting distributive lattice. For constructing a pointwise channel lattice based on the redundancy lattice, we notate the map of functions as shown in Equation (12) and consider the function $k^t\left({\mathbf{S}}_i\right)=T^t\to {\mathbf{S}}_i={\kappa }_i^t$ to obtain the pointwise channel ${\kappa }_i^t$ of a source ${\mathbf{S}}_i$.

$$\begin{array}{cc}\hfill f\langle \mathbf{P}\rangle & =\left\{f\right(\mathbf{x})\mid \mathbf{x}\in \mathbf{P}\},\hfill \\ \hfill f\langle \langle \mathbf{P}\rangle \rangle & =\{f\langle \mathbf{x}\rangle \mid \mathbf{x}\in \mathbf{P}\},\hfill \\ \hfill f\langle \langle \langle \mathbf{P}\rangle \rangle \rangle & =\{f\langle \langle \mathbf{x}\rangle \rangle \mid \mathbf{x}\in \mathbf{P}\}.\hfill \end{array}$$

(12)

The intersections shall correspond to some meet operation and the union to some joint operation on the pointwise channels, as shown in Equation (13), while maintaining the ordering relation of Williams and Beer [5]. This section aims to define suitable meet and joint operations together with a function for their consistent valuation. Each atom ${\alpha }^t,{\beta }^t\in {\mathcal{B}}^t\left(\mathbf{V}\right)$ now represents an expression of channels ${\kappa }^t$ with the operators $\vee /\wedge$, as shown in Appendix A. For example, the element $\{{\mathbf{S}}_{12},{\mathbf{S}}_3\}$ is converted to the expression $({\kappa }_1^t\vee {\kappa }_2^t)\wedge {\kappa }_3^t$.

$${\mathcal{B}}^t\left(\mathbf{V}\right)=\bigwedge \langle \bigvee \langle \langle k^t\langle \langle \langle \mathcal{A}\left(\mathbf{V}\right)\rangle \rangle \rangle \rangle \rangle \rangle .$$

(13)

As seen in Section 3.1, we want to define the joint for a set of channels to be equivalent to their convex hull, matching the Blackwell order. This also ensures that the joint operation is closed on channels.

$$\begin{array}{ccc}\hfill {\kappa }_1^t\vee {\kappa }_2^t& \equiv {\kappa }_1^t\bigsqcup {\kappa }_2^t\hfill & \hfill \left(\mathrm{joint}\mathrm{is}\mathrm{closed}\mathrm{on}\mathrm{channels}\right)\end{array}$$

(14)

Since opposing vectors cancel under the defined equivalence relation, we can use a notion of the Möbius inverse to define the set of vectors spanning a partial decision region ${\alpha }^{\delta t}$ for an atom ${\alpha }^t\in {\mathcal{B}}^t\left(\mathbf{V}\right)$, as shown in Equation (15), written as a recursive block matrix and using the strict down-set of the ordering based on the underlying redundancy lattice.

$${\alpha }^{\delta t}\equiv \left[\begin{array}{cc}{\alpha }^t& -\left[\begin{array}{c}{\beta }^{\delta t}\mid {\beta }^t\in \dot{\downarrow }{\alpha }^t\end{array}\right]\end{array}\right]$$

(15)

The definition of the meet operator (∧) and the extension of the joint operator (∨) from channels to atoms is now obtained from the constraint that the partial contribution for the joint of two incomparable atoms $({\alpha }^t,{\beta }^t\in {\mathcal{B}}^t\left(\mathbf{V}\right),{\alpha }^t\vee {\beta }^t\nsim {\alpha }^t\mathrm{and}{\alpha }^t\vee {\beta }^t\nsim {\beta }^t)$ shall be zero, as shown in Equation (16).

$${\alpha }^t\vee {\beta }^t\nsim {\alpha }^t\mathrm{and}{\alpha }^t\vee {\beta }^t\nsim {\beta }^t\Rightarrow {({\alpha }^t\vee {\beta }^t)}^{\delta t}\equiv \left[\begin{array}{c}0\\ 0\end{array}\right]$$

(16)

This creates the desired inclusion–exclusion principle and results in the equivalences of the meet for two and three atoms, as shown in Equation (17). Their resulting partial channels (${\alpha }^{\delta t}$) correspond to the set of vectors spanning the desired unique and shared decision regions of Figure 6.

$$\begin{array}{cc}\hfill {\alpha }^t\wedge {\beta }^t& \sim \left[\begin{array}{ccc}{\alpha }^t& {\beta }^t& -{\alpha }^t\vee {\beta }^t\end{array}\right]\hfill \end{array}$$

(17a)

$$\begin{array}{cc}\hfill {\alpha }^t\wedge ({\beta }^t\wedge {\gamma }^t)& \sim \left[\begin{array}{ccccccc}{\alpha }^t& {\beta }^t& {\gamma }^t& -{\alpha }^t\vee {\beta }^t& -{\alpha }^t\vee {\gamma }^t& -{\beta }^t\vee {\gamma }^t& {\alpha }^t\vee {\beta }^t\vee {\gamma }^t\end{array}\right]\hfill \end{array}$$

(17b)

From their construction, the meet and joint operators provide a distributive lattice for a set of channels under the defined equivalence relation as shown in Appendix B by satisfying idempotency, commutativity, associativity, absorption and distributivity. This can be used to define a corresponding ordering relation (Equation (18)).

$${\alpha }^t⪯{\beta }^t\equiv {\alpha }^t\wedge {\beta }^t\sim {\alpha }^t\iff {\alpha }^t\vee {\beta }^t\sim {\beta }^t$$

(18)

To obtain a consistent valuation of this lattice, we consider a function $f\left({\alpha }^t\right)$, as shown in Equation (19). First, this function has to be invariant under the defined equivalence relation, and second, it has to match the ordering of the constructed lattice.

$$f\left({\alpha }^t\right)=\sum _{\overrightarrow{v}\in {\alpha }^t}r\left(\overrightarrow{v}\right)\mathrm{where}r\mathrm{is}\mathit{convex}\mathrm{and}\mathrm{satisfies}r(\ell \overrightarrow{v})=\ell r\left(\overrightarrow{v}\right)\mathrm{and}r\left(\left[\begin{array}{c}\ell \\ \ell \end{array}\right]\right)=0$$

(19)

The function f shall apply a (convex) function $r\left(\overrightarrow{v}\right)$ to each vector of the matrix of an atom $\overrightarrow{v}\in {\alpha }^t$. The function is invariant under the equivalence relation (∼, Equation (10)):

• Zero vectors do not affect the quantification: $r\left(\left[\begin{array}{c}0\\ 0\end{array}\right]\right)=0$ 
• The structure of f ensures invariance under reordering columns: $f\left({\kappa }^t\right)=f\left({\kappa }^t\mathbf{P}\right)$ 
• The property $r(\ell \overrightarrow{v})=\ell r\left(\overrightarrow{v}\right)$ with $\ell \in \mathbb{R}$ ensures invariance under splitting/merging columns of identical likelihood ratios:

  $$f\left(\left[(1+\ell ){\overrightarrow{v}}_1\right]\right)=(1+\ell )r\left({\overrightarrow{v}}_1\right)=r\left({\overrightarrow{v}}_1\right)+\ell r\left({\overrightarrow{v}}_1\right)=f\left([{\overrightarrow{v}}_1\ell {\overrightarrow{v}}_1]\right)$$

The function f is a consistent valuation of the ordering relation (⪯, Equation (18)) from the constructed lattice:

• The convexity of r ensures that the quantification $f\left({\alpha }^t\right)$ is a valuation as shown in Appendix C: ${\beta }^t⪯{\alpha }^t\Rightarrow f\left({\beta }^t\right)\le f\left({\alpha }^t\right)$ 
• The function f provides a sum-rule: $f({\alpha }^t\wedge {\beta }^t)=f\left(\left[\begin{array}{ccc}{\alpha }^t& {\beta }^t& -{\alpha }^t\vee {\beta }^t\end{array}\right]\right)=f\left({\alpha }^t\right)+f\left({\beta }^t\right)-f({\alpha }^t\vee {\beta }^t)$ 
• The function f quantifies the bottom element correctly: $f(\perp )=r\left(\left[\begin{array}{c}\ell \\ \ell \end{array}\right]\right)=0$

A parameterized function that forms a consistent lattice valuation with $0\le p\le 1$ and that will be used in Section 3.3 is shown in Equation (20) (the convexity of $r_p$ is shown in Appendix D).

$$\begin{array}{cc}\hfill f_p\left({\alpha }^t\right)& =\sum _{\overrightarrow{v}\in {\alpha }^t}{r}_p\left(\overrightarrow{v}\right)\hfill \end{array}$$

(20a)

$$\begin{array}{cc}\hfill r_p\left(\overrightarrow{v}\right)& =r_p\left(\left[\begin{array}{c}x\\ y\end{array}\right]\right)=xlog\left(\frac{x}{px+(1-p)y}\right)\hfill \end{array}$$

(20b)

This section demonstrated the construction of a distributive lattice and its consistent valuation, resulting in an algebra as shown in Equation (9).

3.3. Decomposing Mutual Information

This section demonstrates that mutual information is the expected value of a consistent valuation for the constructed pointwise lattices and discusses the resulting algebra. To show this, we define the parameter p and pointwise channel ${\kappa }_i^t$ for the consistent valuation (Equation (20)) using a one-vs-rest encoding (Equation (21)).

$$\begin{array}{ccc}\hfill p& =P_{(T=t)}\hfill & \hfill \left(\mathrm{parameter}\right)\\ \hfill {\kappa }_i^t& =\left[\begin{array}{c}{P}_{\left({\mathbf{S}}_i\right|T=t)}\\ P_{\left({\mathbf{S}}_i\right|T\ne t)}\end{array}\right]=\left[\begin{array}{cccc}{x}_1& x_2& \dots & x_m\\ y_1& y_2& \dots & y_m\end{array}\right]\hfill & \hfill \left(\mathrm{binary}\mathrm{input}\mathrm{channel}\right)\end{array}$$

(21)

The expected value of the resulting valuation in Equation (20) is equivalent to the definition of mutual information, as shown in Equation (22). Therefore, we can interpret mutual information as being the expected value of quantifying the reachable decision regions for each state of the target variable that represent a concept of pointwise uncertainty.

$$\begin{array}{cc}\hfill I(T;{\mathbf{S}}_i)& =\sum _{s\in {\mathcal{S}}_i}\sum _{t\in \mathcal{T}}{P}_{({\mathbf{S}}_i,T)}(s,t)log\left(\frac{P_{({\mathbf{S}}_i,T)}(s,t)}{P_{{\mathbf{S}}_i}\left(s\right)P_T\left(t\right)}\right)\hfill \end{array}$$

(22a)

$$\begin{array}{cc}\hfill & ={\mathbb{E}}_T\left[\sum _{s\in {\mathcal{S}}_i}\underset{x_j}{\underbrace{P_{\left({\mathbf{S}}_i\right|T=t)}\left(s\right)}}log\left(\frac{\stackrel{x_j}{\overbrace{P_{\left({\mathbf{S}}_i\right|T=t)}\left(s\right)}}}{\underset{p}{\underbrace{P_{(T=t)}}}\underset{x_j}{\underbrace{P_{\left({\mathbf{S}}_i\right|T=t)}\left(s\right)}}+\underset{1-p}{\underbrace{(1-P_{(T=t)})}}\underset{y_j}{\underbrace{P_{\left({\mathbf{S}}_i\right|T\ne t)}\left(s\right)}}}\right)\right]\hfill \end{array}$$

(22b)

The expected value for a set of consistent lattice valuations corresponds to a weighted sum such that the resulting lattice remains consistent. Therefore, we can combine the pointwise lattices to extend the definition of mutual information for meet and joint elements, which we will think of as intersections and unions. Let $\alpha$ represent an expression of sources with the operators ∨ and ∧. Then, we can obtain its valuation from the pointwise lattices using the function ${\widehat{I}}_T$, as shown in Equation (23). Notice that we do not define the operators for random variables but only use the notation for selecting the corresponding element on the underlying pointwise lattices. For example, we write $\alpha =({\mathbf{S}}_{12}\wedge {\mathbf{S}}_3)\vee {\mathbf{S}}_4$ to refer to the pointwise atom ${\alpha }^t=({\kappa }_{12}^t\wedge {\kappa }_3^t)\vee {\kappa }_4^t$ on each pointwise lattice.

The special case of atoms that consist of a single source corresponds by construction to the definition of mutual information. However, we propose normalizing the measure, as shown in Equation (23), to capture a degree of inclusion between zero and one. This is possible for discrete variables and will lead to an easier intuition for the later definition of bi-valuations and product spaces by ensuring the same output range for these measures. As a possible interpretation for the special role of the target variable, we like to think of T as the considered origin of information within the system, which then propagates through channels to other variables.

$$\begin{array}{ccccc}\hfill {\widehat{I}}_T\left(\alpha \right)& \equiv \frac{{\mathbb{E}}_T\left[f_{P_T\left(t\right)}\left({\alpha }^t\right)\right]}{{\mathbb{E}}_T\left[f_{P_T\left(t\right)}\left(\top \right)\right]}\hfill & \hfill & =\frac{{\mathbb{E}}_T\left[f_{P_T\left(t\right)}\left({\alpha }^t\right)\right]}{H\left(T\right)}\hfill & \\ \hfill {\widehat{I}}_T\left(T\right)& =1\hfill & \hfill & =\frac{H\left(T\right)}{H\left(T\right)}\hfill \\ \hfill {\widehat{I}}_T\left({\mathbf{S}}_i\right)& ={\widehat{I}}_T(T\wedge {\mathbf{S}}_i)\hfill & \hfill & =\frac{I(T;{\mathbf{S}}_i)}{H\left(T\right)}\hfill \end{array}$$

(23)

We obtain the following algebra with the bi-valuation ${\widehat{I}}_T\left([\alpha ;\beta ]\right)$ that quantifies a degree of inclusion from $\alpha$ within the context of $\beta$. We can think of ${\widehat{I}}_T\left([\alpha ;\beta ]\right)$ as asking how much of the information from $\beta$ about T is shared with $\alpha$.

$$\begin{array}{ccc}\hfill {\widehat{I}}_T(\alpha \vee \beta )& ={\widehat{I}}_T\left(\alpha \right)+{\widehat{I}}_T\left(\beta \right)-{\widehat{I}}_T(\alpha \wedge \beta )\hfill & \hfill \left(\mathrm{Sum}\mathrm{rule}\right)\end{array}$$

(24a)

$$\begin{array}{ccc}\hfill {\widehat{I}}_T\left([\alpha ;\beta ]\right)& \equiv \frac{{\widehat{I}}_T(\alpha \wedge \beta )}{{\widehat{I}}_T\left(\beta \right)}\hfill & \hfill \left(\mathrm{Bi}\text{-}\mathrm{Valuation}\right)\end{array}$$

(24b)

$$\begin{array}{ccc}\hfill {\widehat{I}}_T\left([\alpha \vee \beta ;\gamma ]\right)& ={\widehat{I}}_T\left([\alpha ;\gamma ]\right)+{\widehat{I}}_T\left([\beta ;\gamma ]\right)-{\widehat{I}}_T\left([\alpha \wedge \beta ;\gamma ]\right)\hfill & \hfill \left(\mathrm{Conditioned}\mathrm{sum}\mathrm{rule}\right)\end{array}$$

(24c)

$$\begin{array}{ccc}\hfill {\widehat{I}}_T\left([\beta \wedge \gamma ;\alpha ]\right)& ={\widehat{I}}_T\left([\gamma ;\alpha \wedge \beta ]\right)·{\widehat{I}}_T\left([\beta ;\alpha ]\right)\hfill & \hfill \left(\mathrm{Product}\mathrm{rule}\right)\end{array}$$

(24d)

$$\begin{array}{ccc}\hfill {\widehat{I}}_T\left([\beta ;\alpha \wedge \gamma ]\right)& =\frac{{\widehat{I}}_T\left([\gamma ;\alpha \wedge \beta ]\right)·{\widehat{I}}_T\left([\beta ;\alpha ]\right)}{{\widehat{I}}_T\left([\gamma ;\alpha ]\right)}\hfill & \hfill \left(\mathrm{Bayes}’ \mathrm{Theorem}\right)\end{array}$$

(24e)

Since the definitions satisfy an inclusion–exclusion principle, we obtain the interpretation of classical measures as proposed by Williams and Beer [5]: conditional mutual information $I(T;V_1\mid V_2)$ measures the unique contribution of $V_1$ plus its synergy with $V_2$, and interaction information $I(T;V_1;V_2)$ measures the difference between synergy and shared information, which explains its possible negativity.

As highlighted by Knuth [13], the lattice product (the Cartesian product with ordering $(\alpha ;\beta )⪯(\tau ;\upsilon )\iff \alpha ⪯\tau \mathrm{and}\beta ⪯\upsilon$) can be valuated using a product rule to maintain consistency with the ordering of the individual lattices. This creates an opportunity to define information product spaces for multiple reference variables. Since we normalized the measures, the valuation of the product space will also be normalized to the range from zero to one. The subscript notation $T_1\times T_2$ shall indicate the product of the lattice constructed for $T_1$ with the product of the lattice constructed for $T_2$.

$$\begin{array}{ccc}\hfill {\widehat{I}}_{(T_1\times T_2)}\left((\alpha ;\beta )\right)& ={\widehat{I}}_{T_1}\left(\alpha \right)·{\widehat{I}}_{T_2}\left(\beta \right)\hfill & \hfill \left(\mathrm{Valuation}\mathrm{Product}\mathrm{rule}\right)\end{array}$$

(25a)

$$\begin{array}{ccc}\hfill {\widehat{I}}_{(T_1\times T_2)}\left(([\alpha ;\tau ];[\beta ;\upsilon ])\right)& ={\widehat{I}}_{T_1}\left([\alpha ;\beta ]\right)·{\widehat{I}}_{T_2}\left([\tau ;\upsilon ]\right)\hfill & \hfill \left(\mathrm{Bi}\text{-}\mathrm{Valuation}\mathrm{Product}\mathrm{rule}\right)\end{array}$$

(25b)

The lattice product is distributive over the joint for disjoint elements [13], which leads to the equivalence in Equation (26). Unfortunately, it appears that only the bottom element is disjoint with other atoms in the constructed lattice.

$$\forall t:{\alpha }^t\wedge {\beta }^t\sim \perp \Rightarrow {\widehat{I}}_{(T_1\times T_2)}\left((\alpha \vee \beta ;\tau )\right)={\widehat{I}}_{(T_1\times T_2)}((\alpha ;\tau )\vee (\beta ;\tau ))$$

(26)

Finally, we would like to provide an intuition for this approach based on possible operational scenarios:

• Consider having characterized four radio links and obtained the conditional distributions $P_{V_1|T}$, $P_{(V_2,V_3)|T}$ and $P_{V_4|T}$. We are interested in their joint channel capacity; however, lack the required joint distribution. In this case, we can use their joint ${sup}_{P_T\left(t\right)}{\widehat{I}}_T({\mathbf{S}}_1\vee {\mathbf{S}}_{23}\vee {\mathbf{S}}_4)$ to obtain a (pointwise) lower bound on their joint channel capacity.
• Consider having two datasets $\{T_1,V_1,V_2,V_3\}$ and $\{T_2,V_2,V_3,V_4\}$ that provide different types of labels ($T_x$) and associated features ($V_y$), where some events were recorded in both datasets. In such cases, one may choose to study the cases $T_1\to (V_1,V_2,V_3)$, $T_2\to (V_2,V_3,V_4)$ and $(T_1,T_2)\to (V_1,V_2,V_3,V_4)$ for events appearing in both datasets, which could then be combined into a product lattice ${\widehat{I}}_{(T_1\times T_2\times (T_1,T_2))}$.

4. Applications

This section focuses on applications of the obtained measure from Section 3.3. We first apply the meet operator to the redundancy lattice for constructing a PID. Since an atom of the redundancy lattice $\alpha \in \mathcal{A}\left(\mathbf{V}\right)$ corresponds to a set of sources for which the shared information shall be measured, we use the notation $\bigwedge \alpha$ to obtain an expression for the function ${\widehat{I}}_T$. Section 4.2 additionally utilizes the properties of a Markov chain to demonstrate how the flow of partial information can be traced through system models.

4.1. Partial Information Decomposition

Based on Section 3.3, we can define a measure of shared information $\widehat{I}(\alpha ;T)$ for the elements of the redundancy lattice $\alpha \in \mathcal{A}\left(\mathbf{V}\right)$ in the framework of Williams and Beer [5], as shown in Equation (27). The measure satisfies the three axioms of Williams and Beer [5] (commutativity from the equivalence relation and structure of $f_p$, monotonicity from being a lattice valuation and self-redundancy from removing the normalization), and the decomposition is non-negative since the joint channel ${\kappa }_{12}^t$ is superior to the joint of two channels ${\kappa }_1^t\vee {\kappa }_2^t$ for all $t\in \mathcal{T}$. The partial contribution ${\widehat{I}}^{\delta }(\alpha ;T)$ corresponds to the expected value of the quantified partial decision regions ${\alpha }^{\delta t}$.

This provides the interpretation of Section 3.1, where combining the partial contributions of the up-set corresponds to the expected value of quantifying the decision regions that are lost when losing the variable, while combining the partial contributions of the down-set corresponds to the expected value of quantifying the accessible decision region from this variable. Additionally, we obtain a pointwise version of the property by Bertschinger et al. [10]: if a variable provides unique information, then there is a way to utilize this information for a reward function to some target variable state. Finally, it can be seen that taking the minimal quantification of the different decision regions as done by Williams and Beer [5] leads to a lack in distinguishing distinct reachable decision regions or, as phrased in the literature: a lack of distinguishing “the same information and the same amount of information” [6,7,8,9].

$$\begin{array}{cc}\hfill \forall \alpha \in \mathcal{A}\left(\mathbf{V}\right),\widehat{I}(\alpha ;T)& ={\widehat{I}}_T\left(\bigwedge \alpha \right)·H\left(T\right),\hfill \end{array}$$

(27a)

$$\begin{array}{cc}\hfill {\widehat{I}}^{\delta }(\alpha ;T)& =\widehat{I}(\alpha ;T)-\sum _{\beta \in \dot{\downarrow }\alpha }{\widehat{I}}^{\delta }(\beta ;T)={\mathbb{E}}_T\left[f_{P_T\left(t\right)}\left({\alpha }^{\delta t}\right)\right]\hfill \end{array}$$

(27b)

An identical definition of $\widehat{I}(\alpha ;T)$ can be obtained only based on the Blackwell order, as shown in Equation (28). Let $\alpha \in \mathcal{A}\left(\mathbf{V}\right)$ be a set of sources and let $T^t$ represent a binary target variable (${\mathcal{T}}^t=\{t,\overline{t}\}$) such that $T^t=t\iff T=t$. We can expand the meet operator used in Equation (27a) using the sum-rule and utilize the distributivity for arriving at the joint of two channels, which matches the Blackwell order (Equation (28b)). We write ${\mathbf{S}}_i{\bigsqcup }_{T^t}{\mathbf{S}}_j$ to refer to the joint of ${\mathbf{S}}_i$ and ${\mathbf{S}}_j$ under the Blackwell order with respect to variable $T^t$. This results in the recursive definition of $i\left(\alpha ;T^t\right)$ that corresponds to the definition of mutual information for a single source (Equation (28a)). This expansion of Equation (27a) is particularly helpful since it eliminates the operators ∧/∨ for a simplified implementation.

$$\begin{array}{cc}\hfill i\left(\left\{{\mathbf{S}}_i\right\};T^t\right)& =\sum _{s\in {\mathcal{S}}_i}{P}_{\left({\mathbf{S}}_i\right|T^t=t)}\left(s\right)log\left(\frac{P_{\left({\mathbf{S}}_i\right|T^t=t)}\left(s\right)}{P_{(T^t=t)}{P}_{\left({\mathbf{S}}_i\right|T^t=t)}\left(s\right)+(1-P_{(T^t=t)})P_{\left({\mathbf{S}}_i\right|T^t\ne t)}\left(s\right)}\right)\hfill \end{array}$$

(28a)

$$\begin{array}{cc}\hfill i\left(\left\{{\mathbf{S}}_i\right\}\cup \beta ;T^t\right)& =i\left(\left\{{\mathbf{S}}_i\right\};T^t\right)+i\left(\beta ;T^t\right)-i\left(\{{\mathbf{S}}_i{\bigsqcup }_{T^t}{\mathbf{S}}_j\mid {\mathbf{S}}_j\in \beta \};T^t\right)\hfill \end{array}$$

(28b)

$$\begin{array}{cc}\hfill \widehat{I}(\alpha ;T)& ={\mathbb{E}}_T\left[i\left(\alpha ,T^t\right)\right]\hfill \end{array}$$

(28c)

Our decomposition is equivalent to the measures of Bertschinger et al. [10], Griffith and Koch [11] and Williams and Beer [5] in two special cases:

• For a binary target variable $\mathcal{T}=\{t,\overline{t}\}$ with two observable variables $V_1$ and $V_2$, our approach is identical to Bertschinger et al. [10] and Griffith and Koch [11] since ${\kappa }_1\bigsqcup {\kappa }_2\sim {\kappa }_1^t\vee {\kappa }_2^t\sim {\kappa }_1^{\overline{t}}\vee {\kappa }_2^{\overline{t}}$. Beyond binary target variables, the resulting definitions differ due to the pointwise construction (see Appendix E). 
• If from a pointwise perspective ($T^t$), some variable is Blackwell superior to the other (not necessarily the same each time), then our method is identical to Williams and Beer [5] since the defined meet operation will equal their minimum ${\kappa }_1^t\bigsqcup {\kappa }_2^t\sim {\kappa }_2^t\Rightarrow f_p\left({\kappa }_1^t\right)\le f_p\left({\kappa }_2^t\right)\Rightarrow min(f_p\left({\kappa }_1^t\right),f_p\left({\kappa }_2^t\right))=f_p({\kappa }_1^t\wedge {\kappa }_2^t)=f_p\left({\kappa }_1^t\right)$ and equivalently for the function $i(\alpha ,T^t)$.

A decomposition of typical examples can be found in Appendix E. We also provide an implementation of the PID based on our approach [18].

4.2. Information Flow Analysis

Due to the achieved inclusion–exclusion principle, the data processing inequality of mutual information and the achieved non-negativity of partial information for an arbitrary number of variables, it is possible to trace the flow of information through Markov chains. The measure ${\widehat{I}}_T$ appears suitable for this analysis due to the chaining properties of the underlying pointwise channels that are quantified. The analysis can be applied among others for analyzing communication networks or designing data processing systems.

The flow of information in Markov chains has been studied by Niu and Quinn [19], who considered chaining individual variables $X_1\to X_2\to \dots \to X_n$ and performed a decomposition on $\mathbf{V}=\{X_1,X_2,\dots ,X_n\}$. In contrast to this, we consider Markov chains that map sets of random variables from one step to the next. In this case, it is possible to perform an information decomposition at each step of the Markov chain and identify how the partial information components propagate from one set of variables to the next.

Let $T\to \mathbf{V}\to \mathbf{Q}$ be a Markov chain with the atoms $\alpha \in \mathcal{A}\left(\mathbf{V}\right)$ and $\beta \in \mathcal{A}\left(\mathbf{Q}\right)$, through which we trace the flow of partial information from $\alpha$ to $\beta$ about T. We can measure the shared information between both atoms $\alpha$ and $\beta$, as shown in Equation (29a), to obtain how much information their cumulative components share ${\widehat{J}}^{\cap \to \cap }(\alpha \to \beta ;T)$. Similar to the PID, we remove the normalization for the self-redundancy axiom. To identify how much of the cumulative information of $\beta$ is obtained from the partial information of $\alpha$, we subtract the strict down-set of $\alpha$ on the lattice ($\mathcal{A}\left(\mathbf{V}\right),\preccurlyeq$) as shown in Equation (29b) to obtain ${\widehat{J}}^{\delta \to \cap }(\alpha \to \beta ;T)$. To compute how much of the partial information of $\alpha$ is shared with the partial contribution of $\beta$, we similarly remove the flow from the partial information of $\alpha$ into the strict down-set of $\beta$ on the lattice ($\mathcal{A}\left(\mathbf{Q}\right),\preccurlyeq$), as shown in Equation (29c), to obtain ${\widehat{J}}^{\delta \to \delta }(\alpha \to \beta ;T)$. This can be used to trace the origin of information for each atom $\beta \in \mathcal{A}\left(\mathbf{Q}\right)$ to the previous elements $\alpha \in \mathcal{A}\left(\mathbf{V}\right)$.

The approach is not limited to one step and can be extended for tracing the flow through Markov chains of arbitrary length ${\widehat{J}}^{\delta \to \delta \to \delta \dots }(\alpha \to \beta \to \gamma \dots ;T)$. However, we only trace one step in this demonstration for simplicity.

$$\begin{array}{cc}\hfill {\widehat{J}}^{\cap \to \cap }(\alpha \to \beta ;T)& ={\widehat{I}}_T(\bigwedge \alpha \wedge \bigwedge \beta )·H\left(T\right)\hfill \end{array}$$

(29a)

$$\begin{array}{cc}\hfill {\widehat{J}}^{\delta \to \cap }(\alpha \to \beta ;T)& ={\widehat{J}}^{\cap \to \cap }(\alpha \to \beta ;T)-\sum _{\gamma \in \dot{\downarrow }\alpha }{\widehat{J}}^{\delta \to \cap }(\gamma \to \beta ;T)\hfill \end{array}$$

(29b)

$$\begin{array}{cc}\hfill {\widehat{J}}^{\delta \to \delta }(\alpha \to \beta ;T)& ={\widehat{J}}^{\delta \to \cap }(\alpha \to \beta ;T)-\sum _{\gamma \in \dot{\downarrow }\beta }{\widehat{J}}_T^{\delta \to \delta }(\alpha \to \gamma ;T)\hfill \end{array}$$

(29c)

We demonstrate the Information Flow Analysis using a full-adder as a small logic circuit with the input variables $\mathbf{V}=\{A,B,C_{\mathrm{in}}\}$ and the output $\mathbf{T}=\{S,C_{\mathrm{out}}\}$ as shown in Equation (30). Any ideal implementation of this computation results in the same channel from $\mathbf{V}$ to $\mathbf{T}$. Therefore, they create an identical flow of the partial information from $\mathbf{V}$ to the partial information of $\mathbf{T}$. However, the specific implementation will determine how (over which intermediate representations and paths) the partial information is transported.

$$\begin{array}{cc}\hfill S& =A\oplus B\oplus C_{\mathrm{in}}\hfill \\ \hfill C_{\mathrm{out}}& =A·B+A·C_{\mathrm{in}}+B·C_{\mathrm{in}}\hfill & \\ \hfill & =(A·B)+C_{\mathrm{in}}·(A\oplus B))\hfill & \hfill \left(\mathrm{typical}\mathrm{implementation}\right)\\ \hfill T& =(S,C_{\mathrm{out}})\hfill \end{array}$$

(30)

To make the example more interesting, we consider the implementation of a noisy full-adder, as shown in Figure 7, which allows for bit-flips on wires. We indicate the probability of a bit-flip below each line and imagine this value correlates to the wire length and proximity to others. Now, changing the implementation or even the layout of the same circuit would have an impact on the overall channel.

To perform the analysis, we first have to define the target variable: What it is that we want to measure information about? In this case, we select the joint distribution of the desired computation output T as the target variable and define the noisy computation result to be $\widehat{\mathbf{T}}=\{\widehat{S},{\widehat{C}}_{\mathrm{out}}\}$, as shown in Figure 7. We obtain both variables from their definition by assuming that the input variables $\mathbf{V}$ are independently and uniformly distributed and that bit-flips occurred independently. However, it is worth noting that noise dependencies can be modeled in the joint distribution. This fully characterizes the Markov chain shown in Equation (31).

$$T=(S,C_{\mathrm{out}})\to \mathbf{T}=\{S,C_{\mathrm{out}}\}\to \mathbf{V}=\{A,B,C_{\mathrm{in}}\}\to \mathbf{Q}=\{Q_1,Q_2,Q_3\}\to \mathbf{R}=\{R_1,R_2,R_3\}\to \widehat{\mathbf{T}}=\{\widehat{S},{\widehat{C}}_{\mathrm{out}}\}$$

(31)

We group two variables at each stage to reduce the number of interactions in the visualization. The resulting information flow of the full-adder is shown as a Sankey diagram in Figure 8. Each bar corresponds to the mutual information of a stage in the Markov chain with the input T. The bars’ colors indicate the partial information decomposition of Equation (27). The information flow over one step using Equation (29) is indicated by the width of a line between the partial contributions of two stages. To follow the flow of a particular component over more than one step—for example, to see how the shared information of $\mathbf{T}$ propagates to the shared information of $\widehat{\mathbf{T}}$—the analysis can be performed by tracing multiple steps after extending Equation (29).

The results (Figure 8) show that the decomposition does not attribute unique information to S or $C_{\mathrm{out}}$ about their own joint distribution. The reason for this is shown in Equation (32): both variables provide an equivalent channel for each state of their joint distribution and, thus, an equivalent uncertainty about each state of T. Phrased differently, both variables provide access to the identical decision regions for each state of their joint distribution and can therefore not provide unique information (no advantage for any reward function to any $t\in \mathcal{T}$). If this result feels counter-intuitive, we would also recommend the discussion of the two-bit-copy problem and identity axiom by Finn [9] (p. 16ff.) and Finn and Lizier [20]. The same effect can also be seen when viewing each variable in $\mathbf{V}$ individually (not shown in Figure 8), which causes neither of them to provide unique information on their own about the joint target distribution T.

$$\begin{array}{c}\hfill (T^{(0,0)}\to C_{{}_{\mathrm{out}}})\sim (T^{(0,0)}\to S)\sim \left[\begin{array}{cc}1& 0\\ 3/7& 4/7\end{array}\right]\sim (T^{(1,1)}\to C_{{}_{\mathrm{out}}})\sim (T^{(1,1)}\to S)\\ \hfill (T^{(0,1)}\to C_{{}_{\mathrm{out}}})\sim (T^{(0,1)}\to S)\sim \left[\begin{array}{cc}1& 0\\ 1/5& 4/5\end{array}\right]\sim (T^{(1,0)}\to C_{{}_{\mathrm{out}}})\sim (T^{(1,0)}\to S)\end{array}$$

(32)

The Information Flow Analysis is particularly useful in practice since it can be performed on an arbitrary resolution of the system model to handle its complexity. For example, a small full-adder can be analyzed on the level of gates and wires represented by channels. However, the full-adder is itself a channel that can be used to analyze an n-bit adder on the level of full-adders.

Further applications of the Information Flow Analysis could include the identification of which inputs are most critical for the computational result and where information is being lost. It can also be explored if a notion of robustness in data processing systems could be meaningfully defined based on how much pointwise redundant or shared information of the input $\mathbf{V}$ can be traced to its output $\widehat{\mathbf{T}}$. This might indicate a notion of robustness based on whether or not it is possible to compensate for the unavailability of input sources through a system modification.

Finally, the target variable does not have to be the desired computational outcome as has been done in the demonstration. When thinking about secure multi-party computations, it might be of interest to identify the flow of information from the perspective of some sensitive or private variable (T) to understand the impact of disclosing the final computation result. The possible applications of such an analysis are as diverse as those of information theory.

5. Discussion

We propose the interpretation that the reachable decision regions correspond to different notions of uncertainty about each state of the target variable and that mutual information corresponds to the expected value of quantifying these decision regions. This allows partial information to represent the expected value of quantifying partial decision regions (Equations (27) and (28)), which can be used to attribute mutual information to the visible variables and their interactions (pointwise redundant/shared/unique/synergetic). Since the proposed quantification results in the consistent valuation of a distributive lattice, it creates a novel algebra for mutual information with possible practical applications (Equations (24) and (25)). Finally, the approach allows for tracing information components through Markov chains (Equation (29)), which can be used to model and study a wide range of scenarios. The presented method is directly applicable to discrete and categorical source variables due to their equivalent construction for the reachable decision regions (zonotopes). However, we recommend that the target variable should be categorical since the measure does not consider a notion of distance between target states (achievable estimation proximity). This would be an interesting direction for future work due to its practical application for introducing semantic meaning to sets of variables. An intuitive example is a target variable with 256 states that is used to represent an 8-bit unsigned integer as the computation result. For this reason, we wonder if it is possible to introduce a notion of distance to the analysis such that the classical definition of mutual information becomes the special case for encoding categorical targets.

A recent work by Kolchinsky [21] removes the assumption that an inclusion–exclusion principle relates the intersection and union of information and demands their extractability. This has the disadvantage that a similar algebra or tracing of information would no longer be possible. We tried to address this point by distinguishing the pointwise redundant from the pointwise shared element and also obtain no inclusion–exclusion principle for the pointwise redundancy. We focus in this work on the pointwise shared element due to the resulting properties and operational interpretation from the accessibility and losses of reachable decision regions. Moreover, the relation between the used meet and joint operators provides consistent results from performing the decomposition using the meet operator on a redundancy lattice, as done in this work, or a decomposition using the joint operator on a synergy or loss lattice [22].

Further notions of redundancy and synergy can be studied within this framework if they are extractable, meaning they can be represented by some random variable. Depending on the desired interpretation, the representing variable can be constructed for T and added to the set of visible variables or can be constructed for each pointwise variable $T^t$ and added to the pointwise lattices. We showed an example of the latter in Section 3.1 by adding the pointwise redundant element to the lattice, which we interpret as pointwise extractable components of shared information to quantify the decision regions that can be obtained from each source.

Since our approach satisfies the original axioms of Williams and Beer [5] and results in non-negative partial contributions for an arbitrary number of variables, it cannot satisfy the proposed identity axiom of Harder et al. [8]. This can also be seen by the decomposition examples in Appendix E (

Table A2. Two-bit-copy (TBC) example. The results of our approach differ from Bertschinger et al. [10] and Griffith and Koch [11] since the target variable is not binary.

(a) Distribution				(b) Results
${\mathit{V}}_{\mathbf{1}}$	${\mathit{V}}_{\mathbf{2}}$	$\mathit{T}$	Pr	Method	$\mathit{S}({\mathit{V}}_{\mathbf{1}},{\mathit{V}}_{\mathbf{2}};\mathit{T})$	$\mathit{U}({\mathit{V}}_{\mathbf{1}};\mathit{T})$	$\mathit{U}({\mathit{V}}_{\mathbf{2}};\mathit{T})$	$\mathit{C}({\mathit{V}}_{\mathbf{1}},{\mathit{V}}_{\mathbf{2}};\mathit{T})$
0	0	0	1/4	${\widehat{I}}_T·H\left(T\right)$	1	0	0	1
0	1	1	1/4	$I_{\min }$ [5]	1	0	0	1
1	0	2	1/4	$I^{\pm }$ [20]	1	0	0	1
1	1	3	1/4	$I^{\mathrm{BROJA}}$ [10,11]	0	1	1	0

and Figure A3). We do not consider this a limitation since all four axioms cannot be satisfied without obtaining negative partial information [23], which creates difficulties for interpreting results.

Finally, our approach does not appear to satisfy a target/left chain rule as proposed by Bertschinger et al. [7]. While our approach provides an algebra that can be used to handle multiple target variables, we think that further work on understanding the relations when decomposing with multiple target variables is needed. In particular, it would be helpful for the analysis of complex systems if the flow of already analyzed sub-chains could be reused and their interactions could be predicted.

6. Conclusions

We use the approach of Bertschinger et al. [10] and Griffith and Koch [11] to construct a pointwise partial information decomposition that provides non-negative results for an arbitrary number of variables and target states. The measure obtains an algebra from the resulting lattice structure and enables the analysis of complex multivariate systems in practice. To our knowledge, this is the first alternative to the original measure of Williams and Beer [5] that satisfies their three proposed axioms and results in a non-negative decomposition for an arbitrary number of variables.