Non-Negative Decomposition of Multivariate Information: From Minimum to Blackwell-Specific Information

Abstract

Partial information decompositions (PIDs) aim to categorize how a set of source variables provides information about a target variable redundantly, uniquely, or synergetically. The original proposal for such an analysis used a lattice-based approach and gained significant attention. However, finding a suitable underlying decomposition measure is still an open research question at an arbitrary number of discrete random variables. This work proposes a solution with a non-negative PID that satisfies an inclusion–exclusion relation for any f-information measure. The decomposition is constructed from a pointwise perspective of the target variable to take advantage of the equivalence between the Blackwell and zonogon order in this setting. Zonogons are the Neyman–Pearson region for an indicator variable of each target state, and f-information is the expected value of quantifying its boundary. We prove that the proposed decomposition satisfies the desired axioms and guarantees non-negative partial information results. Moreover, we demonstrate how the obtained decomposition can be transformed between different decomposition lattices and that it directly provides a non-negative decomposition of Rényi-information at a transformed inclusion–exclusion relation. Finally, we highlight that the decomposition behaves differently depending on the information measure used and how it can be used for tracing partial information flows through Markov chains.

1. Introduction

From computer science to neuroscience, we can find the following problem: We would like to know information about a random variable T, called the target, which we cannot observe directly. However, we can obtain information about the target indirectly from another set of variables $\mathbf{V}=\{V_1,\dots ,V_n\}$. We can use information measures to quantify how much information any set of variables provides about the target. When doing so, we can identify the concept of redundancy: For example, if we have two identical variables $V_1=V_2$, then we can use one variable to predict the other and, thus, anything that this other variable can predict. Similarly, we can identify the concept of synergy: For example, if we have two independent variables and a target that corresponds to their XOR operation $T=\left(V_1\mathrm{XOR}{V}_2\right)$, then both variables provide no advantage on their own for predicting the state of T, yet their combination fully determines it. Williams and Beer [1] suggested that it is possible to characterize information as visualized by the Venn diagram for two variables $\mathbf{V}=\{V_1,V_2\}$ in Figure 1a. This decomposition attributes the total information about the target to being redundant, synergetic, or unique to a particular variable. As indicated in Figure 1a by $I(·,T)$, we can quantify three of the areas using information measures. However, this is insufficient to determine the four partial areas that represent the individual contributions. This causes the necessity to extend an information measure to either quantify the amount of redundancy or synergy between a set of variables.

Williams and Beer [1] first proposed a framework for Partial Information Decomposition (PIDs) and found favor by the community [2]. However, the proposed measure of redundancy was criticized for not distinguishing, “the same information and the same amount of information” [3,4,5,6]. The proposal of Williams and Beer [1] focused specifically on mutual information. This work additionally studies the decomposition of any f-information or Rényi-information at discrete random variables. They have significance, among others, in parameter estimations, high-dimensional statistics, hypothesis testing, channel coding, data compression, and privacy analyses [7,8].

1.1. Related Work

Most of the literature focuses on the decomposition of mutual information. Here, many alternative measures have been proposed, but cannot fully replace the original measure of Williams and Beer [1] since they do not provide non-negative results for any $\left|\mathbf{V}\right|$: The special case of bivariate partial information decompositions ($\left|\mathbf{V}\right|=2$) has been well studied, and several non-negative decompositions for the framework of Williams and Beer [1] are known [5,9,10,11,12]. However, each of these decompositions provides negative partial information for $\left|\mathbf{V}\right|>2$. Further research [13,14,15] specifically aimed to define decompositions of mutual information for an arbitrary number of observable variables, but similarly obtained negative partial contributions and the resulting difficulty of interpreting their results. Griffith et al. [3] studied the decomposition of zero-error information and obtained negative partial contributions. Kolchinsky [16] proposed a decomposition framework for an arbitrary number of observable variables that is applicable beyond Shannon information theory, however, where the partial contributions do not sum to the total amount.

In this work, we propose a decomposition measure for replacing the one presented by Williams and Beer [1] while maintaining its desired properties. To achieve this, we combine several concepts from the literature: We use the Blackwell order, a preorder of information channels, for the decomposition and for deriving its operational interpretation, similar to Bertschinger et al. [9] and Kolchinsky [16]. We use its special case for binary input channels, the zonogon order studied by Bertschinger and Rauh [17], to achieve non-negativity at an arbitrary number of variables and provide it with a practical meaning by highlighting its equivalence to the Neyman–Pearson (decision) region. To utilize this special case for a general decomposition, we use the concept of a target pointwise decomposition as demonstrated by Williams and Beer [1] and related to Lizier et al. [18], Finn and Lizier [13], and Ince [14]. Specifically, we use Neyman–Pearson regions of an indicator variable for each target state to define distinct information and quantify pointwise information from its boundary. This allows for the non-negative decomposition of an arbitrary number of variables, where the source and target variables can have an arbitrary finite number of states. Finally, we apply the concepts from measuring on lattices, discussed by Knuth [19], to transform a non-negative decomposition with an inclusion–exclusion relation from one information measure to another while maintaining the decomposition properties.

Remark 1. 

We use the term “target pointwise” or simply “pointwise” within this work to refer to the analysis of each target state individually. This differs from [13,14,18], who use the latter term for the analysis of all joint source–target realizations.

1.2. Contributions

In a recent work [20], we presented a decomposition of mutual information on the redundancy lattice (Figure 1b). This work aims to simplify, generalize, and extend these ideas to make the following contributions to the area of partial information decompositions:

• We propose a representation of distinct uncertainty and distinct information, which is used to demonstrate the unexpected behavior of the measure by Williams and Beer [1] (Section 2.2 and Section 3.1).
• We propose a non-negative decomposition for any f-information measure at an arbitrary number of discrete random variables that satisfies an inclusion–exclusion relation and provides a meaningful operational interpretation (Section 3.2, Section 3.3 and Section 3.5). The decomposition satisfies the original axioms of Williams and Beer [1] (Theorems 3 and 4) and obtains different properties from different information measures (Section 4).
• We demonstrate several transformations of the proposed decomposition: (i) We transform the cumulative measure between different decomposition lattices (Section 3.4). (ii) We demonstrate that the non-negative decomposition of f-information directly provides a non-negative decomposition of Rényi- and Bhattacharyya-information at a transformed inclusion–exclusion relation (Section 3.6).

2. Background

This section aims to provide the required background information and introduce the notation used. Section 2.1 discusses the Blackwell order and its special case at binary targets, the zonogon order, which will be used for operational interpretations and the representation of f-information for its decomposition. Section 2.2 discusses the PID framework of Williams and Beer [1] and the relation between a decomposition based on the redundancy lattice and one based on the synergy lattice. We also demonstrate the unintuitive behavior of the original decomposition measure, which will be resolved by our proposal in Section 3. Section 2.3 provides the considered definitions of f-information, Rényi-information, and Bhattacharyya-information for the later demonstration of transforming decomposition results between measures.

Notation 1 

(Random variables and their distribution). We use the notation T (upper case) to represent a random variable, ranging over the event space $\mathcal{T}$ (calligraphic) containing events $t\in \mathcal{T}$ (lower case) and use the notation $P_T$ (P with subscript) to indicate its probability distribution. The same convention applies to other variables, such as a random variable S with events $s\in \mathcal{S}$ and distribution $P_S$. We indicate the outer product of two probability distributions as $P_S\otimes P_T$, which assigns the product of their marginals $P_S\left(s\right)·P_T\left(t\right)$ to each event $(s,t)$ of the Cartesian product $\mathcal{S}\times \mathcal{T}$. Unless stated otherwise, we use the notation T, S, and V to represent random variables throughout this work.

2.1. Blackwell and Zonogon Order

Definition 1 

(Channel). A channel $\mu =T\to S$ from $\mathcal{T}$ to $\mathcal{S}$ represents a garbling of the input variable T, which results in variable S. Within this work, we represent an information channel μ as a (row) stochastic matrix, where each element is non-negative, and all rows sum to one.

For the context of this work, we consider a variable S to be the observation of the output from an information channel $T\to S$ from the target variable T, such that the corresponding channel can be obtained from their conditional probability distribution, as shown in Equation (1) where $\mathcal{T}=\{t_1,\dots ,t_n\}$ and $\mathcal{S}=\{s_1,\dots ,s_m\}$.

$$\mu =(T\to S)=P_{(S\mid T)}=\left[\begin{array}{ccc}p(s_1\mid t_1)& \cdots & p(s_m\mid t_1)\\ ⋮& \ddots & ⋮\\ p(s_1\mid t_n)& \cdots & p(s_m\mid t_n)\end{array}\right]$$

(1)

Notation 2 

(Binary input channels). Throughout this work, we reserve the symbol κ for binary input channels, meaning κ signals a stochastic matrix of dimension $2\times m$. We use the notation $\overrightarrow{v}\in \kappa$ to indicate a column of this matrix.

Definition 2 

(More informative [17,21]). An information channel ${\mu }_1=T\to S_1$ is more informative than another channel ${\mu }_2=T\to S_2$ if—for any decision problem involving a set of actions $a\in \Omega$ and a reward function $u:(\Omega ,\mathcal{T})\to \mathbb{R}$ that depends on the chosen action and state of the variable T—an agent with access to $S_1$ can always achieve an expected reward at least as high as another agent with access to $S_2$.

Definition 3 

(Blackwell order [17,21]). The Blackwell order is a preorder of channels. A channel ${\mu }_1$ is Blackwell superior to channel ${\mu }_2$, if we can pass its output through a second channel λ to obtain an equivalent channel to ${\mu }_2$, as shown in Equation (2).

$${\mu }_2⊑{\mu }_1⟺{\mu }_2={\mu }_1·\lambda forsome\mathit{stochastic}\mathit{matrix}\lambda$$

(2)

Blackwell [21] showed that a channel is more informative if and only if it is Blackwell superior. Bertschinger and Rauh [17] showed that the Blackwell order does not form a lattice for channels $\mu =T\to S$ if $\left|\mathcal{T}\right|>2$ since the ordering does not provide unique meet and join elements. However, binary target variables $\left|\mathcal{T}\right|=2$ are a special case where the Blackwell order is equivalent to the zonogon order (discussed next) and does form a lattice [17].

Definition 4 

(Zonogon [17]). The zonogon $Z\left(\kappa \right)$ of a binary input channel $\kappa =T\to S$ is defined using the Minkowski sum from the collection of vector segments as shown in Equation (3). The zonogon $Z\left(\kappa \right)$ can similarly be defined as the image of the unit cube ${[0,1]}^{\left|\mathcal{S}\right|}$ under the linear map of κ.

$$Z\left(\kappa \right)\left\{\sum _i{x}_i{\overrightarrow{v}}_i:0\le x_i\le 1,{\overrightarrow{v}}_i\in \kappa \right\}=\left\{\kappa a:a\in {[0,1]}^{\left|\mathcal{S}\right|}\right\}$$

(3)

The zonogon $Z\left(\kappa \right)$ is a centrally symmetric convex polygon, and the set of vectors ${\overrightarrow{v}}_i\in \kappa$ spans its perimeter. Figure 2 shows an example of a binary input channel and its corresponding zonogon.

Definition 5 

(Zonogon sum). The addition of two zonogons corresponds to their Minkowski sum as shown in Equation (4).

$$Z\left({\kappa }_1\right)+Z\left({\kappa }_2\right)\left\{a+b:a\in Z\left({\kappa }_1\right),b\in Z\left({\kappa }_2\right)\right\}=Z\left(\left[\begin{array}{cc}{\kappa }_1& {\kappa }_2\end{array}\right]\right)$$

(4)

Definition 6 

(Zonogon order [17]). A zonogon $Z\left({\kappa }_1\right)$ is zonogon superior to another $Z\left({\kappa }_2\right)$ if and only if $Z\left({\kappa }_2\right)\subseteq Z\left({\kappa }_1\right)$.

Bertschinger and Rauh [17] showed that, for binary input channels, the zonogon order is equivalent to the Blackwell order and forms a lattice (Equation (5)). In the remaining work, we will only discuss binary input channels, such that the orderings of Definitions 2, 3, and 6 are equivalent and can be thought of as zonogons with a subset relation.

$${\kappa }_1⊑{\kappa }_2⟺Z\left({\kappa }_1\right)\subseteq Z\left({\kappa }_2\right)$$

(5)

To obtain an interpretation of what a channel zonogon $Z\left(\kappa \right)$ represents, we can consider a binary decision problem by aiming to predict the state $t\in \mathcal{T}$ of a binary target variable T using the output of channel $\kappa =T\to S$. Any decision strategy $\lambda \in {[0,1]}^{\left|\mathcal{S}\right|\times 2}$ for obtaining a binary prediction $\widehat{T}$ can be fully characterized by its resulting pair of True-Positive Rate (TPR) and False-Positive Rate (FPR), as shown in Equation (6):

$$\kappa ·\lambda =(T\to S\to \widehat{T})=P_{(\widehat{T}\mid T)}=\left[\begin{array}{cc}p(\widehat{T}=t\mid T=t)& p(\widehat{T}\ne t\mid T=t)\\ p(\widehat{T}=t\mid T\ne t)& p(\widehat{T}\ne t\mid T\ne t)\end{array}\right]=\left[\begin{array}{cc}\mathrm{TPR}& 1-\mathrm{TPR}\\ \mathrm{FPR}& 1-\mathrm{FPR}\end{array}\right]$$

(6)

Therefore, a channel zonogon $Z\left(\kappa \right)$ provides the set of all achievable (TPR,FPR)-pairs for a given channel $\kappa$ [20,22]. This can also be seen from Equation (3), where the unit cube $a\in {[0,1]}^{\left|\mathcal{S}\right|}$ represents all possible first columns of the decision strategy $\lambda$. The first column of $\lambda$ fully determines the second since each row has to sum to one. As a result, $\kappa a$ provides the (TPR,FPR)-pair for the decision strategy $\lambda =\left[\begin{array}{cc}a& (1-a)\end{array}\right]$ and the definition of Equation (3) for all achievable (TPR,FPR)-pairs for predicting the state of a binary target variable. Since this will be helpful for operational interpretations, we label the axis of zonogon plots accordingly, as shown in Figure 2. The zonogon ([17], p. 2480) is the Neyman–Pearson region ([7], p. 231).

Definition 7 

(Neyman–Pearson region [7] and decision regions). The Neyman–Pearson region for a binary decision problem is the set of achievable (TPR,FPR)-pairs and can be visualized as shown in Figure 2. The Neyman–Pearson regions underlie the zonogon order, and their boundary can be obtained from the likelihood-ratio test. We refer to subsets of the Neyman–Pearson region as reachable decision regions, or simply decision regions, and the boundary as the zonogon perimeter.

Remark 2. 

Due to the zonogon symmetry, the diagram labels can be swapped (FPR x-axis/TPR y-axis), which changes the interpretation to aiming at a prediction for $T\ne t$.

Notation 3 

(Channel lattice). We use the notation ${\kappa }_1\sqcap {\kappa }_2$ for the meet element of binary input channels under the Blackwell order and ${\kappa }_1\bigsqcup {\kappa }_2$ for their join element. We use the notation ${\top }_{\mathit{BW}}=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]$ for the top element of binary input channels under the Blackwell order and ${\perp }_{\mathit{BW}}=\left[\begin{array}{c}1\\ 1\end{array}\right]$ for the bottom element.

For binary input channels, the meet element of the Blackwell order corresponds to the zonogon intersection $Z({\kappa }_1\sqcap {\kappa }_2)=Z\left({\kappa }_1\right)\cap Z\left({\kappa }_2\right)$ and the join element of the Blackwell order corresponds to the convex hull of their union $Z({\kappa }_1\bigsqcup {\kappa }_2)=\mathrm{Conv}(Z\left({\kappa }_1\right)\cup Z\left({\kappa }_2\right))$. Equation (7) describes this for an arbitrary number of channels.

$$Z\left({\sqcap }_{\kappa \in \mathbf{A}}\kappa \right)=\bigcap _{\kappa \in \mathbf{A}}Z\left(\kappa \right)\mathrm{and}Z\left(\underset{\kappa \in \mathbf{A}}{⨆}\kappa \right)=Conv\left(\bigcup _{\kappa \in \mathbf{A}}Z\left(\kappa \right)\right)$$

(7)

Example 1. 

The remaining work only analyzes indicator variables, so we only need to consider the case $\left|\mathcal{T}\right|=2$ where all presented ordering relations of this section are equivalent and form a lattice.

Figure 3a visualizes a channel $T\stackrel{\kappa }{\to }S$ with $\left|\mathcal{S}\right|=3$. We can use the observations of S for making a prediction $\widehat{T}$ about T. For example, we predict that T is in its first state with probability $w_1$ if S is in its first state, with probability $w_2$ if S is in its second state, and with probability $w_3$ if S is in its third state. These randomized decision strategies can be noted as stochastic matrix λ shown in Figure 3a. The resulting TPR and FPR of this decision strategy is obtained from the weighted sum of these parameters ($w_1$, $w_2$, and $w_3$) with the vectors in κ. Each decision strategy corresponds to a point within the zonogon, since the probabilities are constrained by $w_1,w_2,w_3\in [0,1]$ and the resulting zonogon is the Neyman–Pearson region.

Figure 3b visualizes an example for the discussed ordering relations, where all observable variables have two states: $|{\mathcal{S}}_i|=2$ where $i\in \{1,2,3\}$. The zonogon/Neyman–Pearson region corresponding to variable $S_3$ is fully contained within the others ($Z\left({\kappa }_3\right)\subseteq Z\left({\kappa }_1\right)$ and $Z\left({\kappa }_3\right)\subseteq Z\left({\kappa }_2\right)$). Therefore, we can say that $S_3$ is Blackwell inferior (Definition 3) and less informative (Definition 2) than $S_1$ and $S_2$ about T. Practically, this means that we can construct an equivalent variable to $S_3$ by garbling $S_1$ or $S_2$ and that, for any sequence of actions based on $S_3$ and any reward function with dependence on T, we can achieve an expected reward at least as a high by acting based on $S_1$ or $S_2$ instead. The variables $S_1$ and $S_2$ are incomparable to the zonogon order, Blackwell order, and informativity order, since the Neyman–Pearson region of one is not fully contained in the other.

The zonogon shown in Figure 3a corresponds to the join under the zonogon order, Blackwell order, and informativity order of $S_1$ and $S_2$ in Figure 3b about T. For binary targets, this distribution can directly be obtained from the convex hull of their Neyman–Pearson regions and corresponds to a valid joint distribution for $(T,S_1,S_2)$. All other joint distributions are either equivalent or superior to it. When doing this on indicator variables for $\left|\mathcal{T}\right|>2$, then the obtained joint distributions for each $t\in \mathcal{T}$ may not combine into a specific valid overall joint distribution.

2.2. Partial Information Decomposition

The commonly used framework for PIDs was introduced by Williams and Beer [1]. A PID is computed with respect to a particular random variable that we would like to know information about, called the target, and tries to identify from which variables that we have access to, called visible variables, we obtain this information. Therefore, this section considers sets of variables that represent their joint distribution.

Notation 4. 

Throughout this work, we use the notation T for the target variable and $\mathbf{V}=\{V_1,\dots ,V_n\}$ for the set of visible variables. We use the notation $\mathcal{P}\left(\mathbf{V}\right)$ for the power set of $\mathbf{V}$ and ${\mathcal{P}}_1\left(\mathbf{V}\right)=\mathcal{P}\left(\mathbf{V}\right)\setminus \varnothing$ for its power set without the empty set.

Definition 8 

(Sources, atoms [1]).

• A source ${\mathbf{S}}_i\in {\mathcal{P}}_1\left(\mathbf{V}\right)$ is a non-empty set of visible variables.
• An atom $\alpha \in \mathcal{A}\left(\mathbf{V}\right)$ is a set of sources constructed by Equation (8).

  $$\mathcal{A}\left(\mathbf{V}\right)=\{\alpha \in \mathcal{P}\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right):\forall {\mathbf{S}}_a,{\mathbf{S}}_b\in \alpha ,{\mathbf{S}}_a\cancel{\subset }{\mathbf{S}}_b\},$$

  (8)

The filter used for obtaining the set of atoms (Equation (8)) removes sets that would be equivalent to other elements. This is required for obtaining a lattice from the following two ordering relations:

Definition 9 

(Redundancy/gain lattice [1]). The redundancy lattice $\left(\mathcal{A}\right(\mathbf{V}),\preccurlyeq )$ is obtained by applying the ordering relation of Equation (9) to all atoms $\alpha ,\beta \in \mathcal{A}\left(\mathbf{V}\right)$.

$$\alpha \preccurlyeq \beta ⟺\forall {\mathbf{S}}_b\in \beta ,\exists {\mathbf{S}}_a\in \alpha ,{\mathbf{S}}_a\subseteq {\mathbf{S}}_b$$

(9)

The redundancy lattice for three visible variables is visualized in Figure 4a. On this lattice, we can think of an atom as representing the information that can be obtained from all of its sources about the target T (their redundancy or informational intersection). For example, the atom $\alpha =\{\{V_1,V_2\},\{V_1,V_3\}\}$ represents on the redundancy lattice the information that is contained in both $(V_1,V_2)$ and $(V_1,V_3)$ about T. Since both sources in $\alpha$ provide the information of $V_1$, their redundancy contains at least this information, and the atom $\beta =\left\{\left\{V_1\right\}\right\}$ is considered its predecessor. Therefore, the ordering indicates an informational subset relation for the redundancy of atoms, and the information that is represented by an atom increases as we move up. The up-set of an atom $\alpha$ on the redundancy lattice indicates the information that is lost when losing all of its sources. Considering the example from above, if we lose access to $\left\{V_1\left(\mathrm{or}\right)V_2\right\}$ and $\left\{V_1\left(\mathrm{or}\right)V_3\right\}$, then we lose access to all atoms in the up-set of $\alpha =\{\{V_1,V_2\},\{V_1,V_3\}\}$.

Definition 10 

(Synergy/loss lattice [23]). The synergy lattice $\left(\mathcal{A}\right(\mathbf{V}),⪯)$ is obtained by applying the ordering relation of Equation (10) to all atoms $\alpha ,\beta \in \mathcal{A}\left(\mathbf{V}\right)$.

$$\alpha ⪯\beta ⟺\forall {\mathbf{S}}_b\in \beta ,\exists {\mathbf{S}}_a\in \alpha ,{\mathbf{S}}_b\subseteq {\mathbf{S}}_a$$

(10)

The synergy lattice for three visible variables is visualized in Figure 4b. On this lattice, we can think of an atom as representing the information that is contained in neither of its sources (information outside their union). For example, the atom $\alpha =\{\{V_1,V_2\},\{V_1,V_3\}\}$ represents on the synergy lattice the information that is obtained from neither $(V_1,V_2)$ nor $(V_1,V_3)$ about T. The ordering again indicates their expected subset relation: the information that is obtained from neither $\left\{V_1\left(\mathrm{and}\right)V_2\right\}$ nor $\left\{V_1\left(\mathrm{and}\right)V_3\right\}$ is fully contained in the information that cannot be obtained from $\beta =\left\{\left\{V_1\right\}\right\}$, and thus, $\alpha$ is a predecessor of $\beta$.

With an intuition for both ordering relations in mind, we can see how the filter in the construction of atoms (Equation (8)) removes sets that would be equivalent to another atom: the set $\{\{V_1,V_2\},\left\{V_1\right\}\}$ is removed from the power set of sources since it would be equivalent to the atom $\left\{\left\{V_1\right\}\right\}$ under the ordering of the redundancy lattice and equivalent to the atom $\left\{\{V_1,V_2\}\right\}$ under the ordering of the synergy lattice. Using Definition 11, one can similarly define the atoms of the decomposition lattices from the power set of sources without the equivalence relation.

Definition 11. 

We define equivalence relations for sets of sources under the redundancy and synergy order:

$$\begin{array}{cc}\hfill \mathit{Redundancy}\mathit{order}:(\alpha \simeq \beta )& ⟺(\alpha \preccurlyeq \beta \mathit{and}\beta \preccurlyeq \alpha )\hfill \end{array}$$

(11a)

$$\begin{array}{cc}\hfill \mathit{Synergy}\mathit{order}:(\alpha \cong \beta )& ⟺(\alpha ⪯\beta \mathit{and}\beta ⪯\alpha )\hfill \end{array}$$

(11b)

We use the notation $\mathbf{A}\{\cong \}\mathbf{B}$ to indicate that two sets of atoms are equal when comparing their contained atoms with respect to equivalence under the synergy order.

Notation 5 

(Redundancy/synergy lattices). We use the notation $\left(\mathcal{A}\right(V),⋎,⋏)$ for the join and meet operators on the redundancy lattice, and $\left(\mathcal{A}\right(V),\vee ,\wedge )$ for the join and meet operators on the synergy lattice. We use the notation ${\top }_{\mathit{RL}}=\left\{\mathbf{V}\right\}$ for the top and ${\perp }_{\mathit{RL}}=\varnothing$ for the bottom atom on the redundancy lattice, and ${\top }_{\mathit{SL}}=\varnothing$ and ${\perp }_{\mathit{SL}}=\left\{\mathbf{V}\right\}$ for the top and bottom atom on the synergy lattice. For an atom α on the redundancy lattice, we use the notation ${\downarrow }_R\alpha$ for its down-set, ${\dot{\downarrow }}_R\alpha$ for its strict down-set, ${\uparrow }_R\alpha$ for its up-set, ${\dot{\uparrow }}_R\alpha$ for its strict up-set, and ${\alpha }^{{-}_R}$ for its cover set. For an atom α on the synergy lattice, we use the notation ${\downarrow }_S\alpha$ for its down-set, ${\dot{\downarrow }}_S\alpha$ for its strict down-set, ${\uparrow }_S\alpha$ for its up-set, ${\dot{\uparrow }}_S\alpha$ for its strict up-set, and ${\alpha }^{{-}_S}$ for its cover set.

For convenience,

Table 1. Summary of the notation used for the redundancy and synergy lattice.

	Redundancy Order	Synergy Order
Ordering/equivalence	≼/≃	⪯/≅
Join/meet	⋎/⋏	∨/∧
Up-set/strict up-set	${\uparrow }_R$/${\dot{\uparrow }}_R$	${\uparrow }_S$/${\dot{\uparrow }}_S$
Down-set/strict down-set	${\downarrow }_R$/${\dot{\downarrow }}_R$	${\downarrow }_S$/${\dot{\downarrow }}_S$
Cover-set	${\alpha }^{{-}_R}$	${\alpha }^{{-}_S}$
Top/bottom	${\top }_{\mathit{RL}}=\left\{\mathbf{V}\right\}$/${\perp }_{\mathit{RL}}=\varnothing$	${\top }_{\mathit{SL}}=\varnothing$/${\perp }_{\mathit{SL}}=\left\{\mathbf{V}\right\}$

provides a summary of the notation used.

The redundant, unique, or synergetic information (partial contributions) can be calculated based on either lattice. They are obtained by quantifying each atom of the redundancy or synergy lattice with a cumulative measure that increases as we move up in the lattice. The partial contributions are then obtained in a second step from a Möbius inverse.

Definition 12 

([Cumulative] redundancy measure [1]). A redundancy measure $I_{\cap }(\alpha ;T)$ is a function that assigns a real value to each atom of the redundancy lattice. It is interpreted as a cumulative information measure that quantifies the redundancy between all sources $\mathbf{S}\in \alpha$ of an atom $\alpha \in \mathcal{A}\left(\mathbf{V}\right)$ about the target T.

Definition 13 

([Cumulative] loss measure [23]). A loss measure $I_{\cup }(\alpha ;T)$ is a function that assigns a real value to each atom of the synergy lattice. It is interpreted as a cumulative measure that quantifies the information about T that is provided by neither of the sources $\mathbf{S}\in \alpha$ of an atom $\alpha \in \mathcal{A}\left(\mathbf{V}\right)$.

To ensure that a redundancy measure actually captures the desired concept of redundancy, Williams and Beer [1] defined three axioms that a measure $I_{\cap }$ should satisfy. For the synergy lattice, we consider the equivalent axioms discussed by Chicharro and Panzeri [23]:

Axiom 1 

(Commutativity [1,23]). Invariance in the order of sources (σ permuting the order of indices):

$$\begin{array}{cc}\hfill I_{\cap }(\{{\mathbf{S}}_1,\dots ,{\mathbf{S}}_i\};T)& =I_{\cap }(\{{\mathbf{S}}_{\sigma \left(1\right)},\dots ,{\mathbf{S}}_{\sigma \left(i\right)}\};T)\hfill \\ \hfill I_{\cup }(\{{\mathbf{S}}_1,\dots ,{\mathbf{S}}_i\};T)& =I_{\cup }(\{{\mathbf{S}}_{\sigma \left(1\right)},\dots ,{\mathbf{S}}_{\sigma \left(i\right)}\};T)\hfill \end{array}$$

Axiom 2 

(Monotonicity [1,23]). Additional sources can only decrease redundant information. Additional sources can only decrease the information that is in neither source.

$$\begin{array}{cc}\hfill I_{\cap }(\{{\mathbf{S}}_1,\dots ,{\mathbf{S}}_{i-1}\};T)& \ge I_{\cap }(\{{\mathbf{S}}_1,\dots ,{\mathbf{S}}_i\};T)\hfill \\ \hfill I_{\cup }(\{{\mathbf{S}}_1,\dots ,{\mathbf{S}}_{i-1}\};T)& \ge I_{\cup }(\{{\mathbf{S}}_1,\dots ,{\mathbf{S}}_i\};T)\hfill \end{array}$$

Axiom 3 

(Self-redundancy [1,23]). For a single source, redundancy equals mutual information. For a single source, the information loss equals the difference between the total available mutual information and the mutual information of the considered source with the target.

$$\begin{array}{c}\hfill I_{\cap }(\left\{{\mathbf{S}}_i\right\};T)=I({\mathbf{S}}_i;T)\mathit{and}{I}_{\cup }(\left\{{\mathbf{S}}_i\right\};T)=I(\mathbf{V};T)-I({\mathbf{S}}_i;T)\end{array}$$

The first axiom states that an atom’s redundancy and information loss should not depend on the order of its sources. The second axiom states that adding sources to an atom can only decrease the redundancy of all sources (redundancy lattice) and decrease the information from neither source (synergy lattice). The third axiom binds the measures to be consistent with mutual information and ensures that the bottom element of both lattices is quantified to zero.

Once a lattice with the corresponding cumulative measure ($I_{\cap }$/$I_{\cup }$) is defined, we can use the Möbius inverse to compute the partial contribution of each atom. This partial information can be visualized as the partial area in a Venn diagram (see Figure 1a) and corresponds to the desired redundant, unique, and synergetic contributions. However, the same atom represents different partial contributions on each lattice: As visualized for the case of two visible variables in Figure 1, the unique information of variable $V_1$ is represented by $\alpha =\left\{\left\{V_1\right\}\right\}$ on the redundancy lattice and by $\beta =\left\{\left\{V_2\right\}\right\}$ on the synergy lattice.

Definition 14 

(Partial information [1,23]). Partial information $\Delta I_{\cap }(\alpha ;T)$ and $\Delta I_{\cup }(\alpha ;T)$ corresponds to the Möbius inverse of its corresponding cumulative measure on the respective lattice.

$$\begin{array}{ccccc}\hfill \mathit{Redundancy}\mathit{lattice}:& \hfill & \hfill \Delta I_{\cap }(\alpha ;T)& =I_{\cap }(\alpha ;T)-\sum _{\beta \in {\dot{\downarrow }}_R\alpha }\Delta I_{\cap }(\beta ;T),\hfill & \end{array}$$

(12a)

$$\begin{array}{cccc}\hfill \mathit{Synergy}\mathit{lattice}:& \hfill & \hfill \Delta I_{\cup }(\alpha ;T)& =I_{\cup }(\alpha ;T)-\sum _{\beta \in {\dot{\downarrow }}_S\alpha }\Delta I_{\cup }(\beta ;T).\hfill \end{array}$$

(12b)

Remark 3. 

Using the Möbius inverse for defining partial information enforces an inclusion–exclusion relation in that all partial information contributions have to sum to the corresponding cumulative measure. Kolchinsky [16] argues that an inclusion–exclusion relation should not be expected to hold for PIDs and proposes an alternative decomposition framework. In this case, the sum of partial contributions (unique/redundant/synergetic information) is no longer expected to sum to the total amount $I(\mathbf{V};T)$.

Property 1 

(Local positivity, non-negativity [1]). A partial information decomposition satisfies non-negativity or local positivity if its partial information contributions are always non-negative, as shown in Equation (13).

$$\forall \alpha \in \mathcal{A}\left(V\right).\Delta I_{\cap }(\alpha ;T)\ge 0\mathit{or}\Delta I_{\cup }(\alpha ;T)\ge 0$$

(13)

The non-negativity property is important if we assume an inclusion–exclusion relation since it states that the unique, redundant, or synergetic information cannot be negative. If an atom $\alpha$ provides a negative partial contribution in the framework of Williams and Beer [1], then this may indicate that we over-counted some information in its down-set.

Remark 4. 

Several additional axioms and properties have been suggested since the original proposal of Williams and Beer [1], such as target monotonicity and the target chain rule [4]. However, this work will only consider the axioms and properties of Williams and Beer [1]. To the best of our knowledge, no other measure since the original proposal (discussed below) has been able to satisfy these properties for an arbitrary number of visible variables while ensuring an inclusion–exclusion relation for their partial contributions.

It is possible to convert between both representations due to a lattice duality:

Definition 15 

(Lattice duality and dual-decompositions [23]). Let $C=(\mathcal{A}\left(\mathbf{V}\right)\setminus \left\{{\perp }_{\mathit{RL}}\right\},\preccurlyeq )$ be a redundancy lattice with associated measure $I_{\cap }$, and let $D=(\mathcal{A}\left(\mathbf{V}\right)\setminus \left\{{\perp }_{\mathit{SL}}\right\},⪯)$ be a synergy lattice with measure $I_{\cup }$; then, the two decompositions are said to be dual if and only if the down-set on one lattice corresponds to the up-set in the other, as shown in Equation (14).

$$\begin{array}{cc}\hfill \forall \alpha \in C,\exists \beta \in D& :\Delta I_{\cap }(\alpha ;T)=\Delta I_{\cup }(\beta ;T)\hfill \end{array}$$

(14a)

$$\begin{array}{cc}\hfill \forall \alpha \in D,\exists \beta \in C& :\Delta I_{\cup }(\alpha ;T)=\Delta I_{\cap }(\beta ;T)\hfill \end{array}$$

(14b)

$$\begin{array}{cc}\hfill \forall \alpha \in C,\exists \beta \in D& :\phantom{\Delta }{I}_{\cap }(\alpha ;T)=\sum _{{\gamma \in \downarrow }_R\alpha }\Delta I_{\cap }(\gamma ;T)=\sum _{{\gamma \in \uparrow }_S\beta }\Delta I_{\cup }(\gamma ;T)\hfill \end{array}$$

(14c)

$$\begin{array}{cc}\hfill \forall \alpha \in D,\exists \beta \in C& :\phantom{\Delta }{I}_{\cup }(\alpha ;T)=\sum _{{\gamma \in \downarrow }_S\alpha }\Delta I_{\cup }(\gamma ;T)=\sum _{{\gamma \in \uparrow }_R\beta }\Delta I_{\cap }(\gamma ;T)\hfill \end{array}$$

(14d)

$$\begin{array}{cc}\hfill I_{\cap }({\perp }_{\mathit{RL}};T)=& I_{\cup }({\perp }_{\mathit{SL}};T)=0=\Delta I_{\cap }({\perp }_{\mathit{RL}};T)=\Delta I_{\cup }({\perp }_{\mathit{SL}};T)\hfill \end{array}$$

(14e)

Williams and Beer [1] proposed $I_{\cap }^{min}$, as shown in Equation (15), to be used as a measure of redundancy and demonstrated that it satisfies the three required axioms and local positivity. They define redundancy (Equation (15b)) as the expected value of the minimum specific information (Equation (15a)).

Remark 5. 

Throughout this work, we use the term “target pointwise information” or simply “pointwise information” to refer to “specific information”. This shall avoid confusion when naming their corresponding binary input channels in Section 3.

$$\begin{array}{cc}\hfill I({\mathbf{S}}_i;T=t)& =\sum _{s\in {\mathcal{S}}_i}p(s\mid t)\left[log\left(\frac{1}{p\left(t\right)}\right)-log\left(\frac{1}{p(t\mid s)}\right)\right]\hfill \end{array}$$

(15a)

$$\begin{array}{cc}\hfill I_{\cap }^{min}({\mathbf{S}}_1,\dots ,{\mathbf{S}}_k;T)& =\sum _{t\in \mathcal{T}}p\left(t\right)\underset{i\in 1..k}{min}I({\mathbf{S}}_i;T=t).\hfill \end{array}$$

(15b)

To the best of our knowledge, this measure is the only existing non-negative decomposition that satisfies all three axioms listed above for an arbitrary number of visible variables while providing an inclusion–exclusion relation of partial information.

However, the measure $I_{\cap }^{min}$ could be criticized for not providing a notion of distinct information due to its use of a pointwise minimum (for each $t\in \mathcal{T}$) over the sources. This leads to the question of distinguishing “the same information and the same amount of information” [3,4,5,6]. We can use the definition through a pointwise minimum (Equation (15)) to construct examples of unexpected behavior: consider, for example, a uniform binary target variable T and two visible variables as the output of the channels visualized in Figure 5. The channels are constructed to be equivalent for both target states and provide access to distinct decision regions while ensuring constant pointwise information $\forall t\in \mathcal{T}:I(V_x,T=t)=0.2$.

Even though our ability to predict the target variable significantly depends on which of the two indicated channel outputs we observe (blue or green in Figure 5, incomparable informativity based on Definition 2), the measure $I_{\cap }^{min}$ concludes full redundancy between them $I(V_1;T)=I_{\cap }^{min}(\{V_1,V_2\};T)=I(V_2,T)=0.2$. We think this behavior is undesired and, as discussed in the literature, caused by an underlying lack of distinguishing the same information. To resolve this issue, we will present a representation of f-information in Section 3.1, which allows the use of all (TPR,FPR)-pairs for each state of the target variable to represent a distinct notion of uncertainty.

2.3. Information Measures

This section discusses two generalizations of mutual information at discrete random variables based on f-divergences and Rényi-divergences [24,25]. While mutual information has interpretational significance in channel coding and data compression, other f-divergences have their significance in parameter estimations, high-dimensional statistics, and hypothesis testing ([7], p. 88), while Rényi-divergences can be found among others in privacy analysis [8]. Finally, we introduce Bhattacharyya information for demonstrating that it is possible to chain decomposition transformations in Section 3.6. All definitions in this section only consider the case of discrete random variables (which is what we need for the context of this work).

Definition 16 

(f-divergence [24]). Let $f:(0,\infty )\to \mathbb{R}$ be a function that satisfies the following three properties:

• f is convex;
• $f\left(1\right)=0$;
• $f\left(z\right)$ is finite for all $z>0$.

By convention, we understand that $f\left(0\right)={lim}_{z\to 0^{+}}f\left(z\right)$ and $0f\left(\frac{0}{0}\right)=0$. For any such function f and two discrete probability distributions P and Q over the event space $\mathcal{X}$, the f-divergence for discrete random variables is defined as shown in Equation (16).

$$D_f(P\Vert Q)\sum _{x\in \mathcal{X}}Q\left(x\right)f\left(\frac{P\left(x\right)}{Q\left(x\right)}\right)={\mathbb{E}}_Q\left[f\left(\frac{P\left(X\right)}{Q\left(X\right)}\right)\right]$$

(16)

Notation 6. 

Throughout this work, we reserve the name f for functions that satisfy the required properties for an f-divergence of Definition 16.

An f-divergence quantifies a notion of dissimilarity between two probability distributions P and Q. Key properties of f-divergences are their non-negativity, their invariance under bijective transformations, and them satisfying a data-processing inequality ([7], p. 89). A list of commonly used f-divergences is shown in

Table 2. Commonly used functions for f-divergences.

Notation	Name	Generator Function
$D_{\mathrm{KL}}$	Kullback-Leiber (KL)-divergence	$f\left(z\right)=zlogz$
$D_{\mathit{TV}}$	Total Variation (TV)	$f\left(z\right)=\frac{1}{2}\left|z-1\right|$
$D_{{\chi }^2}$	${\chi }^2$-divergence	$f\left(z\right)={(z-1)}^2$
$D_{{\mathrm{H}}^2}$	Squared Hellinger distance	$f\left(z\right)={(1-\sqrt{z})}^2$
$D_{\mathrm{LC}}$	Le Cam distance	$f\left(z\right)=\frac{1-z}{2z+2}$
$D_{\mathrm{JS}}$	Jensen–Shannon divergence	$f\left(z\right)=zlog\frac{2z}{z+1}+log\frac{2}{z+1}$
$D_{{\mathrm{H}}_a}$	Hellinger-divergence with $a\in (0,1)\cup (1,\infty )$	$f\left(z\right)=\frac{z^a-1}{a-1}$
$D_{\alpha =a}$	$\alpha$-divergence with $a\in (0,1)\cup (1,\infty )$	$f\left(z\right)=\frac{z^a-1-a(z-1)}{a(a-1)}$

. Notably, the continuation for $a=1$ of both the Hellinger- and $\alpha$-divergence results in the KL-divergence [26].

The generator function of an f-divergence is not unique since $D_{f\left(z\right)}=D_{f\left(z\right)+c(z-1)}$ for a real constant $c\in \mathbb{R}$ ([7], p. 90f). As a result, the considered $\alpha$-divergence is a linear scaling of the Hellinger divergence ($D_{{\mathrm{H}}_a}=a·D_{\alpha =a}$), as shown in Equation (17).

$$\frac{z^a-1}{a-1}+c(z-1)=a·\frac{z^a-1-a(z-1)}{a(a-1)}\mathrm{for}c=-\frac{a}{a-1}$$

(17)

Definition 17 

(f-information [7]). An f-information is defined based on an f-divergence from the joint distribution of two discrete random variables and the product of their marginals, as shown in Equation (18).

$$\begin{array}{cc}\hfill I_f(S;T):& =D_f\left(P_{(S,T)}\Vert P_S\otimes P_T\right)\hfill \\ \hfill & =\sum _{(s,t)\in \mathcal{S}\times \mathcal{T}}{P}_S\left(s\right)·P_T\left(t\right)·f\left(\frac{P_{(S,T)}(s,t)}{P_S\left(s\right)·P_T\left(t\right)}\right)\hfill \\ \hfill & =\sum _{t\in \mathcal{T}}{P}_T\left(t\right)\left[\sum _{s\in \mathcal{S}}{P}_S\left(s\right)·f\left(\frac{P_{S\mid T}(s\mid t)}{P_S\left(s\right)}\right)\right]\hfill \end{array}$$

(18)

Definition 18 

(f-entropy). A notion of f-entropy for a discrete random variable is obtained from the self-information of a variable $H_f\left(T\right)I_f(T;T)$.

Notation 7. 

Using the KL-divergence results in the definition of mutual information and Shannon entropy. Therefore, we use the notation $I_{\mathit{KL}}$ for mutual information (KL-information) and $H_{\mathit{KL}}$ (KL entropy) for the Shannon entropy.

The remaining part of this section will define Rényi- and Bhattacharyya-information to highlight that they can be represented as an invertible transformation of Hellinger-information. This will be used in Section 3.6 to transform the decomposition of Hellinger-information to a decomposition of Rényi- and Bhattacharyya-information.

Remark 6. 

We could similarly choose to represent Rényi-divergence as a transformation of the α-divergence. A liner scaling of the considered f-divergence will, however, not affect our later results (see Section 3.6).

Definition 19 

(Rényi divergence [25]). Let P and Q be two discrete probability distributions over the event space $\mathcal{X}$, then Rényi-divergence $R_a$ is defined as shown in Equation (19) for $a\in (0,1)\cup (1,\infty )$, and extended to $a\in \{0,1,\infty \}$ by continuation.

$$\begin{array}{cc}\hfill R_a(P\Vert Q):& =\frac{1}{a-1}log\left({\mathbb{E}}_Q\left[{\left(\frac{P\left(X\right)}{Q\left(X\right)}\right)}^a\right]\right)\hfill \\ \hfill & =\frac{1}{a-1}log\left(1+(a-1){\mathbb{E}}_Q\left[\frac{{\left(\frac{P\left(X\right)}{Q\left(X\right)}\right)}^a-1}{a-1}\right]\right)\hfill \\ \hfill & =\frac{1}{a-1}log\left(1+(a-1)D_{{\mathrm{H}}_a}(P\Vert Q)\right)\hfill \end{array}$$

(19)

Notably, the continuation of Rényi-divergence for $a=1$ also equals the KL-divergence ([7], p. 116). Rényi-divergence can be expressed as an invertible transformation of the Hellinger-divergence ($D_{{\mathrm{H}}_a}$; see Equation (19)) [26].

Definition 20 

(Rényi-information [7]). Rényi-information is defined equivalent to f-information as shown in Equation (20) and corresponds to an invertible transformation of Hellinger-information ($I_{{\mathrm{H}}_a}$).

$$\begin{array}{cc}\hfill I_{R_a}(S;T):& =R_a\left(P_{(S;T)}\Vert P_S\otimes P_T\right)\hfill \\ \hfill & =\frac{1}{a-1}log\left(1+(a-1)I_{{\mathrm{H}}_a}(S;T)\right)\hfill \end{array}$$

(20)

Finally, we consider the Bhattacharyya distance (Definition 21), which is equivalent to a linear scaling from a special case of Rényi-divergence (Equation (21)) [26]. It is applied, among others, in signal processing [27] and coding theory [28]. The corresponding information measure (Equation (22)) is like its distance, the scaling of a special case of Rényi-information.

Definition 21 

(Bhattacharyya distance [29]). Let P and Q be two discrete probability distributions over the event space $\mathcal{X}$, then the Bhattacharyya distance is defined as shown in Equation (21).

$$\begin{array}{cc}\hfill B(P\Vert Q):& =-log\left(\sum _{x\in \mathcal{X}}\sqrt{P\left(x\right)Q\left(x\right)}\right)\hfill \\ \hfill & =-log\left(\sum _{x\in \mathcal{X}}Q\left(x\right)\sqrt{\frac{P\left(x\right)}{Q\left(x\right)}}\right)\hfill \\ \hfill & =-log\left(1-0.5·{\mathbb{E}}_Q\left[\frac{{\left(\frac{P\left(X\right)}{Q\left(X\right)}\right)}^{0.5}-1}{0.5-1}\right]\right)\hfill \\ \hfill & =-log\left(1-0.5·D_{{\mathrm{H}}_{0.5}}(P\Vert Q)\right)\hfill \\ \hfill & =0.5·R_{0.5}(P\Vert Q)\hfill \end{array}$$

(21)

Definition 22 

(Bhattacharyya-information). Bhattacharyya-information is defined equivalent to f-information as shown in Equation (22).

$$I_{\mathrm{B}}(S;T)B\left(P_{(S,T)}\Vert P_S\otimes P_T\right)=0.5·I_{R_{0.5}}(S;T)$$

(22)

Example 2. 

Consider the channel $T\stackrel{\kappa }{\to }S$ with $\mathcal{T}=\{t_1,t_2\}$ and $\mathcal{S}=\{s_1,s_2\}$. While it will be discussed in more detail in Section 3.1, Equation (23) already indicates that f-information can be interpreted as the expected value of quantifying the boundary of the Neyman–Pearson region for an indicator variable of each target state $t\in \mathcal{T}$. Each state of a source variable $s\in \mathcal{S}$ corresponds to one side/edge of this boundary as discussed in Section 2.1 and visualized in Figure 2. Therefore, the sum over $s\in \mathcal{S}$ corresponds to the sum of quantifying each edge of the zonogon by some function, which is only parameterized by the distribution of the indicator variable for t. This function satisfies a triangle inequality (Corollary A1), and the total boundary is non-negative (Theorem 2 discussed later). Therefore, we can vaguely think of pointwise f-information as quantifying the length of the boundary of the Neyman–Pearson region or zonogon perimeter to give an oversimplified intuition.

$$\begin{array}{cc}\hfill I_f(S;T)& =\sum _{t\in \mathcal{T}}{P}_T\left(t\right)\underset{\mathrm{pointwise} \mathrm{information} \mathrm{of} \mathrm{an} \mathrm{indicator} \mathrm{variable} T=t}{\underbrace{\left[\sum _{s\in \mathcal{S}}\stackrel{\stackrel{\mathrm{quantifies} \mathrm{each} \mathrm{zonogon} \mathrm{edge}}{︷}}{P_S\left(s\right)·f\left(\frac{P_{S\mid T}(s\mid t)}{P_S\left(s\right)}\right)}\right]}}\hfill \end{array}$$

(23)

Below is a stepwise computation of ${\chi }^2$-information ($f\left(z\right)={(z-1)}^2$) on a small example from this interpretation for the setting of Equation (24).

$$\begin{array}{cc}\hfill \kappa =P_{S\mid T}& =\left[\begin{array}{cc}p(S=s_1\mid T=t_1)& p(S=s_2\mid T=t_1)\\ p(S=s_1\mid T=t_2)& p(S=s_2\mid T=t_2)\end{array}\right]=\left[\begin{array}{cc}0.8& 0.2\\ 0.35& 0.65\end{array}\right]\hfill \end{array}$$

(24a)

$$\begin{array}{cc}\hfill P_T& =\left[\begin{array}{cc}p(T=t_1)& p(T=t_2)\end{array}\right]=\left[\begin{array}{cc}0.4& 0.6\end{array}\right]\phantom{\left[\begin{array}{c}0\\ 0\end{array}\right]}\hfill \end{array}$$

(24b)

Since $\left|\mathcal{T}\right|=2$, we compute the pointwise information for two indicator variables as shown in Figure 6. Since each state $s\in \mathcal{S}$ corresponds to one edge of the zonogon, we compute them individually. Notice that the quantification of each vector $v_{s_i}$ can be expressed as a function that is only parameterized by the distribution of the indicator variable. The total zonogon perimeter is quantified as the sum of each of its edges, which equals pointwise information. In this particular case, we obtain $0.292653$ for the total boundary on the indicator of $t_1$ and $0.130068$ for the total boundary on the indicator of $t_2$. The expected information corresponds to the expected value of these pointwise quantifications and provides the final result (Equation (25)).

$$I_{{\chi }^2}(S;T)=p(T=t_1)·0.292653+p(T=t_2)·0.130068=0.195102$$

(25)

3. Decomposition Methodology

To construct a partial information decomposition in the framework of Williams and Beer [1], we only have to define a cumulative redundancy measure ($I_{\cap }$) or cumulative loss measure ($I_{\cup }$). However, doing this requires a meaningful definition of when information is the same. Therefore, Section 3.1 presents an interpretation of f-information that enables a representation of distinct information. Specifically, we demonstrate that pointwise f-information for a target state $t\in \mathcal{T}$ corresponds to the Neyman–Pearson region of its indicator variable, which is quantified by its boundary (zonogon perimeter). This allows for the interpretation that each distinct (TPR,FPR)-pair for predicting a state of the target variable provides a distinct notion of uncertainty. This interpretation of f-information is used in Section 3.2 to construct a partial information decomposition on the synergy lattice under the Blackwell order for each state $t\in \mathcal{T}$ individually. These individual decompositions are then combined into the final result. Therefore, we decompose specific information based on the Blackwell order rather than using its minimum, like Williams and Beer [1]. The resulting operational interpretation is discussed in Section 3.3. Section 3.4 studies the relation between decomposition lattices to derive the dual-decomposition of any f-information on the redundancy lattice in the following Section 3.5 and prove its correctness. We use the obtained decomposition for any f-information in Section 3.6 to transform a Hellinger-information decomposition into a Rényi-information decomposition while maintaining its non-negativity and an inclusion–exclusion relation. To achieve the desired axioms and properties, we combine different aspects of the existing literature:

• Like Bertschinger et al. [9] and Kolchinsky [16], we base the decomposition on the Blackwell order and use this to obtain the operational interpretation of the decomposition.
• Like Williams and Beer [1] and related to Lizier et al. [18], Finn and Lizier [13], and Ince [14], we perform a decomposition from a pointwise perspective, but only for the target variable.
• In a similar manner to how Finn and Lizier [13] used probability mass exclusion to differentiate distinct information, we use Neyman–Pearson regions for each state of a target variable to differentiate distinct information.
• We propose applying the concepts about lattice re-graduations discussed by Knuth [19] to PIDs to transform the decomposition of one information measure to another while maintaining its consistency.

We extend Axiom 3 of Williams and Beer [1] as shown below, to allow binding any information measure to the decomposition.

Axiom 3* (Self-redundancy).For a single source, redundancy $I_{\cap ,*}$ and information loss $I_{\cup ,*}$ correspond to information measure $I_{*}$ as shown below:

$$I_{\cap ,*}(\left\{{\mathbf{S}}_i\right\};T)=I_{*}({\mathbf{S}}_i;T)\mathit{and}{I}_{\cup ,*}(\left\{{\mathbf{S}}_i\right\};T)=I_{*}(\mathbf{V};T)-I_{*}({\mathbf{S}}_i;T)$$

(26)

3.1. Representing f-Information

We begin with an interpretation of f-information, for which we define a pointwise (indicator) variable $\pi (T,t)$ that represents one state of the target variable (Equation (27a)) and construct its pointwise information channel (Definition 23). Then, we define a function $r_f$ based on the generator function of an f-divergence for quantifying (half) the zonogon perimeter of each pointwise information channel (see Figure 2). These perimeter quantifications are pointwise f-information.

Definition 23 

([Target] pointwise binary input channel). We define a target pointwise binary input channel $\kappa (\mathbf{S},T,t)$ from one state of the target variable $t\in \mathcal{T}$ to an information source $\mathbf{S}$ with event space $\mathcal{S}=\{s_1,\cdots ,s_m\}$ as shown in Equation (27b).

$$\begin{array}{cc}\hfill \pi (T,t)& \left\{\begin{array}{cc}1\hfill & \mathit{if}T=t\hfill \\ 0\hfill & \mathit{otherwise}\hfill \end{array}\right.\hfill \end{array}$$

(27a)

$$\begin{array}{cc}\hfill \kappa (\mathbf{S},T,t)\pi (T,t)\to \mathbf{S}& =\left[\begin{array}{ccc}p\left(S=s_1\mid T=t\right)& \cdots & p\left(S=s_m\mid T=t\right)\\ p\left(S=s_1\mid T\ne t\right)& \cdots & p\left(S=s_m\mid T\ne t\right)\end{array}\right]\hfill \end{array}$$

(27b)

Definition 24 

([Target] pointwise f-information).

• We define a function $r_f$ as shown in Equation (28a) to quantify a vector, where $0\le p,x,y\le 1$.
• We define a target pointwise f-information function $i_f$, as shown in Equation (28b), to quantify half the zonogon perimeter for the corresponding pointwise channel $Z\left(\kappa \right(\mathbf{S},T,t\left)\right)$.

$$\begin{array}{cc}\hfill r_f\left(p,\left[\begin{array}{c}x\\ y\end{array}\right]\right):& =\left(px+(1-p)y\right)·f\left(\frac{x}{px+(1-p)y}\right)\hfill \end{array}$$

(28a)

$$\begin{array}{cc}\hfill i_f\left(p,\kappa \right):& =\sum _{\overrightarrow{v}\in \kappa }{r}_f(p,\overrightarrow{v})\hfill \end{array}$$

(28b)

Theorem 1 

(Properties of $r_f$). For a constant $0\le p\le 1$, (1) the function $r_f(p,\overrightarrow{v})$ is convex in $\overrightarrow{v}$, (2) scales linearly in $\overrightarrow{v}$, (3) satisfies a triangle inequality in $\overrightarrow{v}$, (4) quantifies any vector of slope one to zero, and (5) quantifies the zero vector to zero.

Proof. 

• The convexity of $r_f(p,\overrightarrow{v})$ in $\overrightarrow{v}$ is shown separately in Lemma A1 of Appendix A.
• That $r_f(p,\ell \overrightarrow{v})=\ell r_f(p,\overrightarrow{v})$ scales linearly in $\overrightarrow{v}$ can directly be seen from Equation (28a).
• The triangle inequality of $r_f(p,\overrightarrow{v})$ in $\overrightarrow{v}$ is shown separately in Corollary A1 of Appendix A.
• A vector of slope one is quantified to zero $r_f(p,\left[\begin{array}{c}\ell \\ \ell \end{array}\right])=\ell ·f\left(1\right)=0$, since $f\left(1\right)=0$ is a requirement on the generator function of an f-divergence (Definition 16).
• The zero vector is quantified to zero $r_f(p,\left[\begin{array}{c}0\\ 0\end{array}\right])=0·f\left(\frac{0}{0}\right)=0$ by the convention of generator functions for an f-divergence (Definition 16).

□

The function $r_f$ provides the following properties to the pointwise information measure $i_f$.

Theorem 2 

(Properties of $i_f$). The pointwise information measure $i_f$ (1) maintains the ordering relation of the Blackwell order for binary input channels and (2) is non-negative.

Proof. 

• That the function $r_f$ maintains the ordering relation of the Blackwell order on binary input channels is shown separately in Lemma A2 of Appendix A (Equation (29a)).
• The bottom element ${\perp }_{\mathrm{BW}}=\left[\begin{array}{c}1\\ 1\end{array}\right]$ consists of a single vector of slope one, which is quantified to zero by Theorem 1 (Equation (29b)). The combination with Equation (29a) ensures the non-negativity.

$$\begin{array}{cc}\hfill {\kappa }_1⊑{\kappa }_2⟹& i_f(p,{\kappa }_1)\le i_f(p,{\kappa }_2),\hfill \end{array}$$

(29a)

$$\begin{array}{cc}\hfill & i_f(p,{\perp }_{\mathrm{BW}})=0.\hfill \end{array}$$

(29b)

□

An f-information corresponds to the expected value of the target pointwise f-information function defined above (Equation (30)). As a result, we can interpret f-information as the expected value of quantifying (half) the zonogon perimeters for the target pointwise channels $\kappa (\mathbf{S},T,t)$.

$$\begin{array}{cc}\hfill I_f(\mathbf{S};T)& =\sum _{t\in \mathcal{T}}{P}_T\left(t\right)·i_f\left(P_T\left(t\right),\kappa (\mathbf{S},T,t)\right)\hfill \\ \hfill & =\sum _{t\in \mathcal{T}}{P}_T\left(t\right)·\left[\sum _{\overrightarrow{v}\in \kappa (\mathbf{S},T,t)}{r}_f\left(P_T\left(t\right),\overrightarrow{v}\right)\right]\hfill \\ \hfill & =\sum _{t\in \mathcal{T}}{P}_T\left(t\right)·\left[\sum _{s\in \mathcal{S}}{P}_S\left(s\right)·f\left(\frac{P_{S\mid T}(s\mid t)}{P_S\left(s\right)}\right)\right]\hfill \end{array}$$

(30)

3.2. Decomposing f-Information on the Synergy Lattice

With the representation of Section 3.1 in mind, we can define a non-negative partial information decomposition for a set of visible variables $\mathbf{V}=\{V_1,\dots ,V_n\}$ about a target variable T for any f-information. The decomposition is performed from a pointwise perspective, which means that we decompose the pointwise measure $i_f$ on the synergy lattice $\left(\mathcal{A}\right(\mathbf{V}),⪯)$ for each $t\in \mathcal{T}$. The pointwise synergy lattices are then combined using a weighted sum to obtain the decomposition of $I_f$.

We map each atom of the synergy lattice to the join of pointwise channels for its contained sources.

Definition 25 

(From atoms to channels). We define the channel corresponding to an atom $\alpha \in \mathcal{A}\left(\mathbf{V}\right)$ as shown in Equation (31).

$${\kappa }_{\bigsqcup }(\alpha ,T,t)\left\{\begin{array}{cc}{\perp }_{\mathrm{BW}}\hfill & \mathit{if}\alpha =\varnothing \hfill \\ \underset{\mathbf{S}\in \alpha }{⨆}\kappa (\mathbf{S},T,t))\hfill & \mathit{otherwise}\hfill \end{array}\right.$$

(31)

Lemma 1. 

For any set of sources $\alpha ,\beta \in \mathcal{P}\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right)$ and target variable T with state $t\in \mathcal{T}$, the function ${\kappa }_{\bigsqcup }$ maintains the ordering of the synergy lattice under the Blackwell order as shown in Equation (32).

$$\alpha ⪯\beta ⟹{\kappa }_{\bigsqcup }(\beta ,T,t)⊑{\kappa }_{\bigsqcup }(\alpha ,T,t)$$

(32)

Lemma 1 is shown separately in Appendix C. The mapping from Definition 25 provides a lattice that can be quantified using pointwise f-information to construct a cumulative loss measure for its decomposition using the Möbius inverse.

Definition 26 

([Target] pointwise cumulative and partial loss measures). We define the target pointwise cumulative and partial loss functions as shown in Equations (33a) and (33b).

$$\begin{array}{cc}\hfill i_{\cup ,f}(\alpha ,T,t):& =i_f\left(P_T\left(t\right),\kappa (\mathbf{V},T,t)\right)-i_f(P_T\left(t\right),{\kappa }_{\bigsqcup }(\alpha ,T,t))\hfill \end{array}$$

(33a)

$$\begin{array}{cc}\hfill \Delta i_{\cup ,f}(\alpha ,T,t):& =i_{\cup ,f}(\alpha ,T,t)-\sum _{\beta \in {\dot{\downarrow }}_S\alpha }\Delta i_{\cup ,f}(\beta ,T,t)\hfill \end{array}$$

(33b)

The combined cumulative and partial measures are the expected value of their corresponding pointwise measures. This corresponds to combining the pointwise decomposition lattices by a weighted sum.

Definition 27 

(Combined cumulative and partial loss measures). The cumulative loss measure $I_{\cup ,f}$ is defined by Equation (34) and the decomposition result $\Delta I_{\cup ,f}$ by Equation (35).

$$I_{\cup ,f}(\alpha ;T)\sum _{t\in \mathcal{T}}{P}_T\left(t\right)·i_{\cup ,f}(\alpha ,T,t)$$

(34)

$$\begin{array}{cc}\hfill \Delta I_{\cup ,f}(\alpha ;T):& =\sum _{t\in \mathcal{T}}{P}_T\left(t\right)·\Delta i_{\cup ,f}(\alpha ,T,t)\hfill \\ \hfill & =I_{\cup ,f}(\alpha ;T)-\sum _{\beta \in {\dot{\downarrow }}_S\alpha }\Delta I_{\cup ,f}(\beta ;T)\hfill \end{array}$$

(35)

Theorem 3. 

The presented definitions for the pointwise and expected loss measures ($i_{\cup ,f}$ and $I_{\cup ,f}$) provide a non-negative PID on the synergy lattice with an inclusion–exclusion relation that satisfies Axioms 1, 2, and 3* for any f-information measure.

Proof. 

• Axiom 1: The measure $i_{\cup ,f}$ (Equation (33a)) is invariant to permuting the order of sources in $\alpha$, since the join operator of the zonogon order (${⨆}_{\mathbf{S}\in \alpha }$) is. Therefore, also $I_{\cup ,f}$ satisfies Axiom 1.
• Axiom 2: The monotonicity of both $i_{\cup ,f}$ and $I_{\cup ,f}$ on the synergy lattice is shown separately as Corollary A2 in Appendix C.
• Axiom 3*: For a single source, $i_{\cup ,f}$ equals the pointwise information loss by definition (see Equations (26), (28b), and (33a)). Therefore, $I_{\cup ,f}$ satisfies Axiom 3*.
• Non-negativity: The non-negativity of $\Delta i_{\cup ,f}$ and $\Delta I_{\cup ,f}$ is shown separately as Lemma A8 in Appendix C.

□

3.3. Operational Interpretation

From a pointwise perspective ($\left|\mathcal{T}\right|=2$), there always exists a dependency between the sources for which the synergy of this state becomes zero. This dependence corresponds, by definition, to the join of their channels. This is helpful for the operational interpretation in the following paragraph since, individually, each pointwise synergy becomes fully volatile to the dependence between the sources. There may not exist a dependency between the sources for which the expected synergy becomes zero for $\left|\mathcal{T}\right|>2$. However, each decision region that is quantified as synergetic becomes inaccessible at some dependence between the sources.

The decomposition obtains the operational interpretation that, if a variable provides pointwise unique information, then there exists a unique decision region for some $t\in \mathcal{T}$ that this variable provides access to. Moreover, if a set of variables provides synergetic information, then a decision region for some $t\in \mathcal{T}$ may become inaccessible if the dependence between the variables changes. Due to the equivalence of the zonogon and Blackwell order for binary input variables, these interpretations can also be transferred to a set of actions $a\in \Omega$ and a pointwise reward function $u(a,\pi (T,t\left)\right)$, which only depends on one state of the target variable $\pi (T,t)$ (see Section 2.1): If a variable provides unique information, then it provides an advantage for some set of actions and pointwise reward function, while synergy indicates that the advantage for some pointwise reward function is based on the dependence between variables.

The implication of the interpretation does not hold in the other direction, which we will also highlight in the example of $I_{\cup ,\mathit{TV}}$ in Section 4.1. Finally, the definition of the Blackwell order through the chaining of channels (Equation (2)) highlights its suitability for tracing the flows of information in Markov chains (see Section 4.2).

Remark 7. 

The operational interpretation can be strengthened further such that the implication between accessible regions and partial information holds in both directions by revising Lemmas A1 and A2 with a strictly convex generator function to obtain ${\kappa }_1⊏{\kappa }_2⟹i_f(p,{\kappa }_1)<i_f(p,{\kappa }_2)$.

3.4. Decomposition Duality

A non-negative decomposition on the synergy lattice raises the question about its dual-decomposition on the redundancy lattice. Unfortunately, the definition of decomposition duality (Definition 15 [23]) does not specify the mapping between atoms to easily construct dual-decompositions. Therefore, this section discusses how the redundancy and synergy lattice are related by identifying operators that transform one lattice into the other. This transformation can then be used to refine the definition of decomposition duality and, correspondingly, transforms the cumulative measure between lattices.

Definition 28. 

We define two functions: The function $\Xi :\mathcal{P}\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right)\to \mathcal{P}\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right)$ provides the atom with complement sources, and the function $\Psi :\mathcal{P}\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right)\to \mathcal{P}\left({\mathcal{P}}_1\left(\mathbf{V}\right)\right)$ is the n-ary Cartesian product. We indicate the i-th source of an atom as $\alpha \left[i\right]$ and indicate some variable within the i-th source as $x_i$.

$$\begin{array}{cc}\hfill \Xi \left(\alpha \right)& \left\{\begin{array}{cc}\varnothing \hfill & \mathit{if}\alpha =\left\{\mathbf{V}\right\}\hfill \\ \left\{\mathbf{V}\right\}\hfill & \mathit{if}\alpha =\varnothing \hfill \\ \left\{\mathbf{V}\setminus \left\{\mathbf{S}\right\}:\mathbf{S}\in \alpha \right\}\hfill & \mathit{otherwise}\hfill \end{array}\right.\hfill \\ \hfill \Psi \left(\alpha \right)& \left\{\begin{array}{cc}\{\varnothing \}\hfill & \mathit{if}\alpha =\varnothing \hfill \\ \left\{\left\{x_1,\cdots ,x_m\right\}:x_1\in \alpha \left[1\right],\cdots ,x_m\in \alpha \left[m\right]\right\}\hfill & \mathit{otherwise},\mathit{where}m=\left|\alpha \right|\hfill \end{array}\right.\hfill \end{array}$$

(36)

Example 3. 

For an example of these functions, let $\mathbf{V}=\{V_1,V_2,V_3,V_4\}$ and $\alpha =\{\left\{V_1\right\},\{V_2,V_3\}\}$:

$$\begin{array}{cc}\hfill \Xi \left(\alpha \right)& =\left\{\{\{V_2,V_3,V_4\},\{V_1,V_4\}\}\right\}\hfill \\ \hfill \Xi (\Xi (\alpha \left)\right)& =\{\left\{V_1\right\},\{V_2,V_3\}\}=\alpha \hfill \\ \hfill \Psi \left(\alpha \right)& =\{\{V_1,V_2\},\{V_1,V_3\}\}\hfill \\ \hfill \Psi (\Psi (\alpha \left)\right)& =\{\left\{V_1\right\},\{V_1,V_3\},\{V_2,V_1\},\{V_2,V_3\}\}\hfill \\ \hfill & \simeq \{\left\{V_1\right\},\{V_2,V_1\},\{V_2,V_3\}\}\hfill & \hfill (\mathit{since}\left\{V_1\right\}\subseteq \{V_1,V_3\})\\ \hfill & \simeq \{\left\{V_1\right\},\{V_2,V_3\}\}\simeq \alpha \hfill & \hfill (\mathit{since}\left\{V_1\right\}\subseteq \{V_2,V_1\})\\ \hfill \Xi (\Psi (\Psi \left(\alpha \right)\left)\right)& =\{\{V_2,V_3,V_4\},\{V_2,V_4\},\{V_3,V_4\},\{V_1,V_4\}\}\hfill \\ \hfill & \cong \{\{V_2,V_3,V_4\},\{V_3,V_4\},\{V_1,V_4\}\}\hfill & \hfill (\mathit{since}\{V_2,V_4\}\subseteq \{V_2,V_3,V_4\})\\ \hfill & \cong \{\{V_2,V_3,V_4\},\{V_1,V_4\}\}\cong \Xi \left(\alpha \right)\hfill & \hfill (\mathit{since}\{V_3,V_4\}\subseteq \{V_2,V_3,V_4\})\end{array}$$

Lemma 2. 

The function $\Psi (·)$ is a bijection on the redundancy lattice without the bottom element (∅) that reverses its order. Let $\alpha ,\beta \in \mathcal{A}\left(\mathbf{V}\right)\setminus \left\{{\perp }_{\mathit{RL}}\right\}$:

• $\Psi (\Psi (\alpha \left)\right)\simeq \alpha$
• $\alpha \preccurlyeq \beta ⟺\Psi \left(\beta \right)\preccurlyeq \Psi \left(\alpha \right)$

Lemma 3. 

The function $\Xi (·)$ is a bijection that maintains the ordering of atoms between the redundancy and synergy order. Let $\alpha ,\beta \in \mathcal{A}\left(\mathbf{V}\right)$:

• $\alpha =\Xi (\Xi (\alpha \left)\right)$
• $\alpha \preccurlyeq \beta ⟺\Xi \left(\alpha \right)⪯\Xi \left(\beta \right)$

The proofs of Lemmas 2 and 3 are given separately in Appendix D.

Corollary 1. 

Without bottom elements, the redundancy ($\mathcal{A}\left(\mathbf{V}\right)\setminus \left\{{\perp }_{\mathit{RL}}\right\},\preccurlyeq$) and synergy lattice ($\mathcal{A}\left(\mathbf{V}\right)\setminus \left\{{\perp }_{\mathit{SL}}\right\},⪯$) are related, as shown below with $\alpha ,\beta \in \mathcal{A}\left(\mathbf{V}\right)\setminus \left\{{\perp }_{\mathit{RL}}\right\}$:

$$\begin{array}{cc}\hfill \alpha \preccurlyeq \beta & ⟺\Xi (\Psi (\beta \left)\right)⪯\Xi (\Psi (\alpha \left)\right)\hfill \end{array}$$

(37a)

$$\begin{array}{cc}\hfill \Xi (\Psi (\alpha ⋏\beta \left)\right)& \cong \Xi (\Psi (\alpha \left)\right)\vee \Xi (\Psi (\beta \left)\right)\hfill \end{array}$$

(37b)

$$\begin{array}{cc}\hfill \Xi (\Psi (\alpha ⋎\beta \left)\right)& \cong \Xi (\Psi (\alpha \left)\right)\wedge \Xi (\Psi (\beta \left)\right)\hfill \end{array}$$

(37c)

$$\begin{array}{cc}\hfill \{\Xi (\Psi \left(\beta \right)):\beta \in {\downarrow }_R\alpha \}& {\{\cong \}\uparrow }_S\Xi (\Psi \left(\alpha \right))\hfill \end{array}$$

(37d)

$$\begin{array}{cc}\hfill \{\Xi (\Psi \left(\beta \right)):\beta \in {\uparrow }_R\alpha \}& {\{\cong \}\downarrow }_S\Xi (\Psi \left(\alpha \right))\hfill \end{array}$$

(37e)

Proof. 

Follows directly from Lemma 2 and 3. □

Figure 7 visualizes the relations from the introduced operators to provide an intuition. Applying the function $\Psi$ to all atoms is equal to reversing the redundancy order, while applying the function $\Xi$ to all atoms is equal to swapping the ordering relation used (synergy/redundancy order).

With these definitions in place, we can refine the definition of decomposition duality:

Lemma 4 

(Decomposition duality). A redundancy- and synergy-based information decomposition is pointwise dual if, for all $\alpha \in \mathcal{A}\left(\mathbf{V}\right)\setminus \left\{{\perp }_{\mathit{RL}}\right\}$:

$$\begin{array}{cc}\hfill \Delta i_{\cap ,f}(\alpha ,T,t)& =\Delta i_{\cup ,f}(\Xi (\Psi \left(\alpha \right)),T,t)\hfill \\ \hfill \Delta i_{\cap ,f}({\perp }_{\mathit{RL}},T,t)& =0=\Delta i_{\cup ,f}({\perp }_{\mathit{SL}},T,t)\hfill \end{array}$$

(38)

A redundancy- and synergy-based information decomposition is dual if, for all $\alpha \in \mathcal{A}\left(\mathbf{V}\right)\setminus \left\{{\perp }_{\mathit{RL}}\right\}$:

$$\begin{array}{cc}\hfill \Delta I_{\cap ,f}(\alpha ;T)& =\Delta I_{\cup ,f}(\Xi (\Psi \left(\alpha \right));T)\hfill \\ \hfill \Delta I_{\cap ,f}({\perp }_{\mathit{RL}};T)& =0=\Delta I_{\cup ,f}({\perp }_{\mathit{SL}};T)\hfill \end{array}$$

(39)

The proof of Lemma 4 is shown separately in Appendix D. To convert a decomposition from the synergy lattice into its dual-decomposition on to the redundancy lattice, the following relation is particularly useful. It states that, on the synergy lattice, all atoms are either in the up-set of $\Xi (\Psi (\alpha \left)\right)$ or in the down-set of an atom that corresponds to an individual source within $\alpha$.

Lemma 5. 

For $\alpha \in \mathcal{A}\left(\mathbf{V}\right)\setminus \left\{{\perp }_{\mathit{RL}}\right\}$:

$${\mathcal{A}\left(\mathbf{V}\right)\setminus \uparrow }_S\Xi (\Psi \left(\alpha \right))=\bigcup _{{\mathbf{S}}_a\in \alpha }{\downarrow }_S\left\{{\mathbf{S}}_a\right\}$$

(40)

Proof. 

When expanding the definition of up- and down-sets, it can directly be seen from Lemma A9 that both sets provide an exclusive partitioning of all atoms.

$$\begin{array}{cc}\hfill {\uparrow }_S\Xi (\Psi \left(\alpha \right))=& \{\beta \in \mathcal{A}\left(\mathbf{V}\right):\left(\Xi (\Psi (\alpha \left)\right)⪯\beta \right)\}\hfill \\ \hfill \bigcup _{{\mathbf{S}}_a\in \alpha }{\downarrow }_S\left\{{\mathbf{S}}_a\right\}=& \{\beta \in \mathcal{A}\left(\mathbf{V}\right):\left(\exists {\mathbf{S}}_a\in \alpha .\beta ⪯\left\{{\mathbf{S}}_a\right\}\right)\}\hfill \\ \hfill \left(\Xi (\Psi (\alpha \left)\right)⪯\beta \right)& ⟺\neg \left(\exists {\mathbf{S}}_a\in \alpha .\beta ⪯\left\{{\mathbf{S}}_a\right\}\right)\hfill & \hfill (\mathrm{by}\mathrm{Lemma}\mathrm{A}9)\end{array}$$

(41)

□

Figure 8 summarizes and visualizes the required relations for the following transformation of the cumulative measure: (i) The bottom elements of all lattices are mapped to each other and quantified to zero. (ii) The function $\Psi$ reverses the redundancy lattice ($\beta \simeq \Psi \left(\alpha \right)$ such that $\alpha \simeq \Psi \left(\beta \right)$) to relate the down-set of $\alpha$ to the up-set of $\beta$ while ignoring the bottom element. The function $\Xi$ captures the relation between both orderings (${\alpha }^{\prime }\cong \Xi \left(\beta \right)$ such that $\beta \simeq \Xi \left({\alpha }^{\prime }\right)$), to relate the up-set of $\beta$ on the redundancy lattice to the up-set of ${\alpha }^{\prime }$ on the synergy lattice. This provides the desired mapping from the down-set of $\alpha$ on the redundancy lattice to the up-set of ${\alpha }^{\prime }$ on the synergy lattice for duality. Alternatively, we could first transform the down-set of $\alpha$ on the redundancy to the down-set of ${\beta }^{\prime }=\Xi \left(\alpha \right)$ on the synergy lattice, then reverse the synergy order and obtain the same result. (iii) Lemma 5 states that all atoms on the synergy lattice are either in the up-set of $\Xi (\Psi (\alpha \left)\right)$ or in the down-set of $\left\{{\mathbf{S}}_a\right\}$ with ${\mathbf{S}}_a\in \alpha$. The example $\alpha =\{{\mathbf{S}}_3,{\mathbf{S}}_{12}\}$ is visualized in Figure 8, and we encourage the reader to view another example such as $\alpha =\left\{{\mathbf{S}}_{13}\right\}\stackrel{\Psi }{\to }\{{\mathbf{S}}_1,{\mathbf{S}}_3\}\stackrel{\Xi }{\to }\{{\mathbf{S}}_{12},{\mathbf{S}}_{23}\}$.

With these relations in place, we can construct dual-decompositions and prove their correctness.

Lemma 6. 

The pointwise dual-decomposition for the redundancy lattice of a loss measure on the synergy lattice is defined by:

$$i_{\cap ,f}(\alpha ,T,t):=\left\{\begin{array}{cc}0\hfill & \mathit{if}\alpha =\varnothing \hfill \\ i_{\cup ,f}({\top }_{\mathit{SL}},T,t)-\sum _{\beta \in {\mathcal{P}}_1\left(\alpha \right)}{(-1)}^{\left|\beta \right|-1}{i}_{\cup ,f}(\beta ,T,t)\hfill & \mathit{otherwise}\hfill \end{array}\right.$$

(42)

The proof of Lemma 6 is shown separately in Appendix D. This section discussed the relation between four decomposition lattices, which are the redundancy and synergy lattice, as well as their reversed counterparts. Additionally, we demonstrated how this relation can be used to transform a cumulative decomposition measure between them. Decomposition duality enforces each lattice to be consistent with its set-theoretic interpretation. The function $\Psi$ corresponds to taking the set-theoretic complement on the redundancy lattice and, thus, reflects on the cumulative measure by subtracting it from the top atom. The function $\Xi$ corresponds to the relation between the union and intersection and, thus, introduces an inclusion–exclusion principle between their cumulative measures.

3.5. Decomposing f-Information on the Redundancy Lattice

Using the results from Section 3.4, we can now convert the decomposition of Section 3.2 to the redundancy lattice. The conversion can be applied to both the expected or pointwise measure. The partial contributions ($\Delta i_{\cap ,f}$ and $\Delta I_{\cap ,f}$) are obtained from the Möbius inverse.

Lemma 7 

(Dual-decomposition on the redundancy lattice). The definitions of Equation (43) correspond to the dual-decomposition of Definition 26.

$$\begin{array}{cc}\hfill i_{\cap ,f}(\alpha ,T,t)& =\left\{\begin{array}{cc}0\hfill & \mathit{if}\alpha =\varnothing \hfill \\ \sum _{\beta \in {\mathcal{P}}_1\left(\alpha \right)}{(-1)}^{\left|\beta \right|-1}{i}_f(P_T\left(t\right),{\kappa }_{\bigsqcup }(\beta ,T,t))\hfill & \mathit{otherwise}\hfill \end{array}\right.\hfill \end{array}$$

(43a)

$$\begin{array}{cc}\hfill I_{\cap ,f}(\alpha ;T)& =\sum _{t\in \mathcal{T}}{P}_T\left(t\right)·i_{\cap ,f}(\alpha ,T,t)=I_f(\mathbf{V};T)-\sum _{\beta \in {\mathcal{P}}_1\left(\alpha \right)}{(-1)}^{\left|\beta \right|-1}{I}_{\cup ,f}(\beta ;T)\hfill \end{array}$$

(43b)

Proof. 

The duality of the pointwise measure is obtained from Lemma 6 and Definition 26. The duality of the pointwise measure implies the duality of the combined measure. □

The function $i_f(P_T\left(t\right),{\kappa }_{\bigsqcup }(\alpha ,T,t))$ quantifies the convex hull/blackwell join of the Neyman–Pearson regions of its sources and represents a notion of pointwise union information about the target state $t\in \mathcal{T}$. It is used in Equation (33a) to define a pointwise loss measure for the synergy lattice by subtracting it from the total information. As expected, we can see that the corresponding dual-decomposition on the redundancy lattice enforces an inclusion–exclusion relation between our notions of pointwise union information ($i_f(P_T\left(t\right),{\kappa }_{\bigsqcup }(\alpha ,T,t))$) and pointwise intersection information ($i_{\cap ,f}(\alpha ,T,t)$).

Theorem 4. 

The dual-decomposition as defined by Equation (43) provides a non-negative PID, which satisfies an inclusion–exclusion relation and the axioms of Williams and Beer [1] on the redundancy lattice for any f-information.

Proof. 

• Axiom 1: The measure $i_{\cap ,f}$ is invariant to permuting the order of sources in $\alpha$, since the join operator of the zonogon order (${⨆}_{\mathbf{S}\in \alpha }$) is. Therefore, also, $I_{\cap ,f}$ satisfies Axiom 1.
• Non-negativity: The non-negativity of $\Delta i_{\cap ,f}$ is obtained from Lemma 7 and Theorem 3 as shown in Equation (44). The non-negativity of the pointwise measure implies the non-negativity of the combined measure $\Delta I_{\cap ,f}$.

  $$\forall \alpha \in \mathcal{A}\left(\mathbf{V}\right).\Delta i_{\cap ,f}(\alpha ,T,t)=\left\{\begin{array}{cc}0\hfill & \ge 0\mathrm{if}\alpha ={\perp }_{\mathit{RL}}\hfill \\ \Delta i_{\cup ,f}(\Xi (\Phi \left(\alpha \right)),T,t)\hfill & \ge 0\mathrm{otherwise}\hfill \end{array}\right.$$

  (44)
• Axiom 2: Since the cumulative measures $i_{\cap ,f}$ and $I_{\cap ,f}$ correspond to the sum of partial contributions in their down-set, the non-negativity of partial information implies the monotonicity of the cumulative measures.
• Axiom 3*: For a single source, $I_{\cap ,f}$ equals f-information by definition (see Equation (30)). Therefore, $I_{\cap ,f}$ satisfies Axiom 3*.

□

The operational interpretation of Section 3.3 is maintained since the partial contributions are identical between both lattices.

Remark 8. 

The definitions of Equations (34) and (43) satisfy the desired property of Bertschinger et al. [9], who argued that any sensible measure for unique and redundant information should only depend on the marginal distribution of sources.

Remark 9. 

As discussed before [20], it is possible to further split redundancy into two components for extracting the pointwise meet under the Blackwell order (zonogon intersection, first component). The second component of redundancy as defined above contains decision regions that are part of the convex hull, but not the individual channel zonogons (discussed as shared information in [20]). By combining Equation (43) and Lemma A7, we obtain that both components of this split for redundancy are non-negative.

3.6. Decomposing Rényi-Information

Since Rényi-information is an invertible transformation of Hellinger-information and $\alpha$-information, we argue that their decompositions should be consistent. We propose to view the decomposition of Rényi-information as a transformation from an f-information and demonstrate the approach by transferring the Hellinger-information decomposition to a Rényi-information decomposition. Then, we demonstrate that the result is invariant to a linear scaling of the considered f-information, such that the transformation from $\alpha$-information provides identical results. The obtained Rényi-information decomposition is non-negative and satisfies the three axioms proposed by Williams and Beer [1] (see below). However, its inclusion–exclusion relation is based on a transformed addition operator. For transforming the decomposition, we consider Rényi-information to be a re-graduation of Hellinger-information, as shown in Equation (45).

$$\begin{array}{cc}\hfill v_a\left(z\right):& =\frac{1}{a-1}log\left(1+(a-1)z\right)\hfill \end{array}$$

(45a)

$$\begin{array}{cc}\hfill I_{{\mathrm{R}}_a}(\mathbf{S};T)& =v_a\left(I_{{\mathrm{H}}_a}(\mathbf{S};T)\right)\hfill \end{array}$$

(45b)

To maintain consistency when transforming the measure, we also have to transform its operators ([19], p. 6 ff.):

Definition 29 

(Addition of Rényi-information). We define the addition of Rényi-information ${\oplus }_a$ with its corresponding inverse function ${\ominus }_a$ by Equation (46).

$$\begin{array}{cc}\hfill x{\oplus }_{a}y& v_a(v_a^{-1}\left(x\right)+v_a^{-1}\left(y\right))=\frac{log\left(e^{(a-1)x}+e^{(a-1)y}-1\right)}{a-1}\hfill \end{array}$$

(46a)

$$\begin{array}{cc}\hfill x{\ominus }_{a}y& v_a(v_a^{-1}\left(x\right)-v_a^{-1}\left(y\right))=\frac{log\left(e^{(a-1)x}-e^{(a-1)y}+1\right)}{a-1}\hfill \end{array}$$

(46b)

To transform a decomposition of the synergy lattice, we define the cumulative loss measures as shown in Equation (47) and use the transformed operators when computing the Möbius inverse (Equation (48a)) to maintain consistency in the results (Equation (48b)).

Definition 30. 

The cumulative and partial Rényi-information loss measures are defined as transformations of the cumulative and partial Hellinger-information loss measures, as shown in Equations (47) and (48).

$$I_{\cup ,{\mathrm{R}}_a}(\alpha ;T):=v_a\left(I_{\cup ,{\mathrm{H}}_a}(\alpha ;T)\right)$$

(47)

$$\begin{array}{cc}\hfill \Delta I_{\cup ,{\mathrm{R}}_a}(\alpha ;T):=& I_{\cup ,{\mathrm{R}}_a}(\alpha ;T){\ominus }_a\sum _{\beta \in {\dot{\downarrow }}_S\alpha }\Delta I_{\cup ,{\mathrm{R}}_a}(\beta ;T)\mathit{where}:+:={\oplus }_a\hfill \end{array}$$

(48a)

$$\begin{array}{cc}\hfill & =v_a(\Delta I_{\cup ,{\mathrm{H}}_a}(\alpha ;T))\hfill \end{array}$$

(48b)

Remark 10. 

We show in Lemma A11 of Appendix E that re-scaling the original f-information does not affect the resulting decomposition or transformed operators. Therefore, transforming a Hellinger-information decomposition or a α-information decomposition to a Rényi-information decomposition provides identical results.

The operational interpretation presented in Section 3.2 is similarly applicable to partial Rényi-information ($\Delta I_{\cup ,{\mathrm{R}}_a}$, Equation (48b)), since the function $v_a$ satisfies $v_a\left(0\right)=0$ and $x\le 0⟹0\le v_a\left(x\right)$.

Theorem 5. 

The presented definitions for the cumulative loss measure $I_{\cup ,{\mathrm{R}}_a}$ provide a non-negative PID on the synergy lattice with an inclusion–exclusion relation under the transformed addition (Definition 29) that satisfies Axioms 1, 2, and 3* for any Rényi-information measure.

Proof. 

• Axiom 1: $I_{\cup ,{\mathrm{R}}_a}(\alpha ;T)$ is invariant to permuting the order of sources, since $I_{\cup ,{\mathrm{H}}_a}(\mathbf{S};T)$ satisfies Axiom 1 (see Section 3.2).
• Axiom 2: $I_{\cup ,{\mathrm{R}}_a}(\alpha ;T)$ satisfies monotonicity, since $I_{\cup ,{\mathrm{H}}_a}(\mathbf{S};T)$ satisfies Axiom 2 (see Section 3.2) and the transformation function $v_a$ is monotonically increasing for $a\in (0,1)\cup (1,\infty )$.
• Axiom 3*: Since $I_{\cup ,{\mathrm{H}}_a}$ satisfies Axiom 3* (see Section 3.2, Equations (45) and (47)), $I_{\cup ,{\mathrm{R}}_a}$ satisfies the self-redundancy axiom by definition, however, at a transformed operator: $I_{\cup ,{\mathrm{R}}_a}(\left\{{\mathbf{S}}_i\right\};T)=I_{{\mathrm{R}}_a}(\left\{\mathbf{V}\right\};T){\ominus }_a{I}_{{\mathrm{R}}_a}(\left\{{\mathbf{S}}_i\right\};T)$.
• Non-negativity: The decomposition of $I_{\cup ,{\mathrm{R}}_a}$ is non-negative, since $\Delta I_{\cup ,{\mathrm{H}}_a}$ is non-negative (see Section 3.2), the Möbius inverse is computed with transformed operators (Equation (48b)) and the function $v_a$ satisfies $x\le 0⟹0\le v_a\left(x\right)$.

□

Remark 11. 

To obtain an equivalent decomposition of Rényi-information on the redundancy lattice, we can correspondingly transform the dual-decomposition from the redundancy lattice of Hellinger-information as shown in Equation (49). The resulting decomposition will satisfy the non-negativity, the axioms of Williams and Beer [1], and an inclusion–exclusion relation under the transformed operators (Definition 29) for the same reasons described above from Theorem 4.

$$\begin{array}{cc}\hfill I_{\cap ,{\mathrm{R}}_a}(\alpha ;T):=& v_a\left(I_{\cap ,{\mathrm{H}}_a}(\alpha ;T)\right)\hfill \end{array}$$

(49a)

$$\begin{array}{cc}\hfill \Delta I_{\cap ,{\mathrm{R}}_a}(\alpha ;T):=& v_a(\Delta I_{\cap ,{\mathrm{H}}_a}(\alpha ;T))\hfill \end{array}$$

(49b)

Remark 12. 

The relation between the redundancy and synergy lattice can be used for the definition of a bi-valuation [19] in calculations as discussed in [20]. This is also possible for Rényi-information at transformed operators.

When taking the limit of Rényi-information for $a\to 1$, we obtain mutual information ($I_{\mathrm{KL}}$). Since mutual information is also an f-information, we expect its operators in the Möbius inverse to be addition. This is indeed the case (Equation (50)), and the measures will be consistent.

$$\begin{array}{cc}\hfill \underset{a\to 1}{lim}x{\oplus }_{a}y& =x+y\hfill \\ \hfill \underset{a\to 1}{lim}x{\ominus }_{a}y& =x-y\hfill \end{array}$$

(50)

Finally, the decomposition of Bhattacharyya-information can be obtained by re-scaling the decomposition of Rényi-information at $a=0.5$, which causes another transform of the addition operator for the inclusion–exclusion relation.

4. Evaluation

A comparison of the proposed decomposition with other methods of the literature can be found in [20] for mutual information. Therefore, this section first compares different f-information measures for typical decomposition examples and discusses the special case of total variation (TV)-information to explain its distinct behavior. Since we can see larger differences between measures in more complex scenarios, we compare the measures by analyzing the information flows in a Markov chain. We provide the implementation used for both dual-decompositions of f-information and the examples used in this work in [30].

4.1. Partial Information Decomposition

4.1.1. Comparison of Different f-Information Measures

We use the examples discussed by Finn and Lizier [13] to compare different f-information decompositions and add a generic example from [20]. All probability distributions used and their abbreviations can be found in Appendix F. We normalize the decomposition results to the f-entropy of the target variable for the visualization in Figure 9.

Since all results are based on the same framework, they behave similarly for examples that analyze a specific aspect of the decomposition function (XOR, Unq, PwUnq, RdnErr, Tbc, AND). However, it can be observed that the decomposition of total variation (TV) appears to differ from others: (1) In all examples, total variation attributes more information to being redundant than other measures. (2) In the generic example, total variation is the only measure that does not attribute any information to being unique to variable one or synergetic. We discuss the case of total variation in Section 4.1.2 to explain its distinct behavior.

We visualize the zonogons for the generic example in Figure A2, which shall highlight that the implication of the operational interpretation does not hold in the other direction: the existence of partial information implies an advantage for the expected reward towards some state of the target variable, but an advantage for the expected reward towards some state of the target variable does not imply partial information in the example of total variation.

4.1.2. The Special Case of Total Variation

The behavior of total variation appears different compared to other f-information measures (Figure 9). This is due to total variation measuring the perimeter of a zonogon such that the result corresponds to a linear scaling of the maximal (Euclidean) height $h^{*}$ that the zonogon reaches above the diagonal, as visualized in Figure 10.

Remark 13. 

From a cost perspective, the height $h^{*}$ can be interpreted as the performance evaluation of the optimal decision strategy (symmetric point to $P^{*}$ in the lower zonogon half) for a prediction $\widehat{T}$ with minimal expected cost at the cost ratio $\frac{\mathrm{Cos}\mathrm{t}(T=t,\widehat{T}\ne t)-\mathrm{Cos}\mathrm{t}(T=t,\widehat{T}=t)}{\mathrm{Cos}\mathrm{t}(T\ne t,\widehat{T}=t)-\mathrm{Cos}\mathrm{t}(T\ne t,\widehat{T}\ne t)}=\frac{1-P_T\left(t\right)}{P_T\left(t\right)}$ (see Equation (8) of [31]) for each target state individually.

Lemma 8. 

(a)

The pointwise total variation ($i_{\mathit{TV}}$) is a linear scaling of the maximal (Euclidean) height $h^{*}$ that the corresponding zonogon reaches above the diagonal, as visualized in Figure 10 (Equation (51a)).

(b)

For a non-empty set of pointwise channels $\mathbf{A}$, pointwise total variation $i_{\mathit{TV}}$ quantifies the join element to the maximum of its individual channels (Equation (51b)).

(c)

The loss measure $i_{\cup ,\mathit{TV}}$ quantifies the meet for a set of sources on the synergy lattice to their minimum (Equation (51c)).

$$\begin{array}{cc}\hfill i_{\mathit{TV}}(p,\kappa )& =\frac{1-p}{2}\sum _{v\in \kappa }|v_x-v_y|=(1-p)\frac{h^{*}}{\sqrt{2}}\hfill \end{array}$$

(51a)

$$\begin{array}{cc}\hfill i_{\mathit{TV}}(p,\underset{\kappa \in \mathbf{A}}{⨆}\kappa )& =\underset{\kappa \in \mathbf{A}}{max}{i}_{\mathit{TV}}(p,\kappa )\hfill \end{array}$$

(51b)

$$\begin{array}{cc}\hfill i_{\cup ,\mathit{TV}}(\underset{\alpha \in \mathbf{A}}{\bigwedge }\alpha ,T,t)& =\underset{\alpha \in \mathbf{A}}{min}{i}_{\cup ,\mathit{TV}}(\alpha ,T,t)\hfill \end{array}$$

(51c)

Proof. 

The proof of the first two statements (Equations (51a) and (51b)) is provided separately in Appendix G, which imply the third (Equation (51c)) by Definition 26. □

Quantifying the meet element on the synergy lattice to the minimum has the following consequences for total variation: (1) It attributes a minimum amount of synergy, and therefore more information to redundancy than other measures. (2) For each state of the target, at most one variable can provide unique information. In the case of $\left|\mathcal{T}\right|=2$, the pointwise channels are symmetric (see Equation (6)), such that the same variable provides the maximal zonogon height both times. This is the case in the generic example of Figure 9, and the reason why at most one variable can provide unique information in this setting. However, beyond binary targets ($\left|\mathcal{T}\right|>2$), both variables may provide unique information at the same time since different sources can provide the maximal zonogon height for different target states (see the later example in Figure 11).

Remark 14. 

Using the pointwise minimum on the synergy lattice results in a similar structure to the proposed measure of Williams and Beer [1]. However, TV-information is based on a different pointwise measure $i_{TV}$, which displays the same behavior (Equation (51b)), unlike pointwise KL-information.

4.2. Information Flow Analysis

The differences between f-information measures in Section 4.1 appear more visible in complex scenarios. Therefore, this section compares different measures in the information flow analysis of a Markov chain.

Consider a Markov chain ${\mathbf{M}}_1\to {\mathbf{M}}_2\to \cdots \to {\mathbf{M}}_5$, where ${\mathbf{M}}_i=(X_i,Y_i)$ is the joint distribution of two variables. Assume that we are interested in state three, and thus, define $T={\mathbf{M}}_3$ as the target variable. Using the approach described in Section 3, we can compute an information decomposition for each state ${\mathbf{M}}_i$ of the Markov chain with respect to the target. Now, we are additionally interested in how the partial information decomposition from stage ${\mathbf{M}}_i$ propagates into the next ${\mathbf{M}}_{i+1}$, as visualized in Figure 11.

Definition 31 

(Partial information flow). The partial information flow of an atom $\alpha \in \mathcal{A}\left({\mathbf{M}}_i\right)$ into the atom $\beta \in \mathcal{A}\left({\mathbf{M}}_{i+1}\right)$ quantifies the redundancy between the partial contributions of their respective decomposition lattices.

Notation 8. 

We use the notation $I_{\circ ,f}$ with $\circ \in \{\cup ,\cap \}$ to refer to either the loss measure $I_{\cup ,f}$ or redundancy measure $I_{\cap ,f}$. The same applies to the functions $J_{\circ \to \circ ,f}$ and $J_{\Delta \to \circ ,f}$ of Equation (52).

Let $\alpha \in \mathcal{A}\left({\mathbf{M}}_i\right)$ and $\beta \in \mathcal{A}\left({\mathbf{M}}_{i+1}\right)$, then we compute information flows equivalently on the redundancy or synergy lattice as shown in Equation (52). When using a redundancy measure $\circ =\cap$, then the strict down-set of ${\dot{\downarrow }}_{\circ }\alpha$ refers to the strict down-set on its redundancy lattice $(\mathcal{A}\left({\mathbf{M}}_i\right),\preccurlyeq )$, and when using a loss measure $\circ =\cup$, then the strict down-set ${\dot{\downarrow }}_{\circ }\alpha$ refers to the strict down-set on its synergy lattice $(\mathcal{A}\left({\mathbf{M}}_i\right),⪯)$. We obtain the intersection of cumulative measures by quantifying their meet, which is on both lattice equivalent to their union of sources ($J_{\circ \to \circ ,f}$, Equation (52a)). To obtain how much of the partial contribution of $\alpha$ can be found in the cumulative measure of $\beta$ ($J_{\Delta \to \circ ,f}$), we remove the contributions of its down-set (${\dot{\downarrow }}_{\circ }\alpha$ on the lattice for $\mathcal{A}\left({\mathbf{M}}_i\right)$, see Equation (52b)). To finally obtain the flow from the partial contribution of $\alpha$ to the partial contribution of $\beta$ ($J_{\Delta \to \Delta ,f}$), we similarly remove the contributions of the down-set of $\beta$ (${\dot{\downarrow }}_{\circ }\beta$ on the lattice for $\mathcal{A}\left({\mathbf{M}}_{i+1}\right)$, see Equation (52c)). The approach can be extended for tracing information flows over multiple steps; however, we will only trace one step in this example.

$$\begin{array}{cc}\hfill J_{\circ \to \circ ,f}(\alpha ,\beta ,T):=& I_{\circ ,f}(\alpha \cup \beta ;T)\hfill \end{array}$$

(52a)

$$\begin{array}{cc}\hfill J_{\Delta \to \circ ,f}(\alpha ,\beta ,T):=& J_{\circ \to \circ ,f}(\alpha ,\beta ,T):=-\sum _{\gamma \in {\dot{\downarrow }}_{\circ }\alpha }{J}_{\Delta \to \circ ,f}(\gamma ,\beta ,T)\hfill \end{array}$$

(52b)

$$\begin{array}{cc}\hfill J_{\Delta \to \Delta ,f}(\alpha ,\beta ,T)& J_{\Delta \to \circ ,f}(\alpha ,\beta ,T)-\sum _{\gamma \in {\dot{\downarrow }}_{\circ }\beta }{J}_{\Delta \to \Delta ,f}(\alpha ,\gamma ,T)\hfill \end{array}$$

(52c)

Remark 15. 

The resulting partial information flows are equivalent (dual) between the redundancy and loss measure, except for the bottom element since their functionality differs: The flow from or to the bottom element on the redundancy lattice is always zero. In contrast, the flow from or to the bottom element on the synergy lattice quantifies the information gained or lost in the step.

Remark 16. 

The information flow analysis of Rényi- and Bhattacharyya-information can be obtained as a transformation of the information flow from Hellinger-information. Alternatively, the information flow can be computed directly using Equation (52) under the corresponding definition of addition and subtraction for the information measure used.

We randomly generate an initial distribution and each row of a transition matrix under the constraint that at least one value shall be above 0.8 to avoid an information decay that is too rapid through the chain. The specific parameters of the example are shown in Appendix H. The event spaces used are $\mathcal{X}=\{0,1,2\}$ and $\mathcal{Y}=\{0,1\}$ such that $|{\mathcal{M}}_i|=6$. We construct a Markov chain of five steps with the target $T={\mathbf{M}}_3$ and trace each partial information for one step using Equation (52). We visualized the results for KL-, TV-, and ${\chi }^2$-information in Figure 11, and the results for ${\mathrm{H}}^2$-, LC-, and JS-information in Figure A3 of Appendix H.

All results display the expected behavior that the information that ${\mathbf{M}}_i$ provides about ${\mathbf{M}}_3$ increases for $1\le i\le 3$ and decreases for $3\le i\le 5$. The information flow results of KL-, ${\mathrm{H}}^2$-, LC-, and JS-information are conceptually similar. Their main differences appear in the rate at which the information decays and, therefore, how much of the total information we can trace. In contrast, the results of TV- and ${\chi }^2$-information display different behavior, as shown in Figure 11: TV-information indicates significantly more redundancy, and ${\chi }^2$-information displays significantly more synergy than the other measures. Additionally, the decomposition of TV-information contains fever information flows. For example, it is the only analysis that does not show any information flow from ${\mathbf{M}}_2$ into the unique contribution of $Y_3$ or from ${\mathbf{M}}_2$ into the synergy of $(X_3,Y_3)$. This demonstrates that the same decomposition method can obtain different behaviors from different f-divergences.

5. Discussion

Using the Blackwell order to construct pointwise lattices and to decompose pointwise information is motivated from the following three aspects:

• All information measures in Section 2.3 are the expected value of the pointwise information (quantification of the Neyman–Pearson region boundary) for an indicator variable of each target state. Therefore, we argue for acknowledging the “pointwise nature” [13] of these information measures and to decompose them accordingly. A similar argument was made previously by Finn and Lizier [13] for the case of mutual information and motivated their proposed pointwise partial information decomposition.
• The Blackwell order does not form a lattice beyond indicator variables since it does not provide a unique meet or join element for $\left|\mathcal{T}\right|>2$ [17]. However, from a pointwise perspective, the informativity (Definition 2) provides a unique representation of union information. This enables separating the definition of redundant, unique, and synergetic information from a specific information measure, which then only serves for its quantification. We interpret these observations as an indication that the Blackwell order should be used to decompose pointwise information based on indicator variables rather than decomposing the expected information based on the full target distribution.
• We can consider where the alternative approach would lead, if we decomposed the expected information from the full target distribution using the Blackwell order: the decomposition would become identical to the method of Bertschinger et al. [9] and Griffith and Koch [10]. For bivariate examples ($\left|\mathbf{V}\right|=2$), this decomposition [9,10] is non-negative and satisfies an additional property (identity, proposed by Harder et al. [5]). However, the identity property is inconsistent [32] with the axioms of Williams and Beer [1] and non-negativity for $\left|\mathbf{V}\right|>2$. This causes negative partial information when extending the approach to $\left|\mathbf{V}\right|>2$. The identity property also contradicts the conclusion of Finn and Lizier [13] from studying Kelly Gambling that, “information should be regarded as redundant information, regardless of the independence of the information sources” ([13], p. 26). It also contradicts our interpretation of distinct information through distinct decision regions when predicting an indicator variable for some target state. We do not argue that this interpretation should be applicable to the concept of information in general, but acknowledge that this behavior seems present in the information measures studied in this work and construct their decomposition accordingly.

Our critique for the decomposition measure of Williams and Beer [1] focuses on the implication that a less informative variable (Definition 2) about $t\in \mathcal{T}$ provides less pointwise information ($I(S;T=t)$, Equation (15a)): $\kappa ({\mathbf{S}}_1,T,t)⊑\kappa ({\mathbf{S}}_2,T,t)⟹I({\mathbf{S}}_1;T=t)\le I({\mathbf{S}}_2;T=t)$. This implication does not hold in the other direction. Therefore, equal pointwise information does not imply equal informativity and, thus, does not mean being redundant.

We chose to define a notion of pointwise union information based on the join of the Blackwell order since it leads to a meaningful operational interpretation: the convex hull of the pointwise Neyman–Pearson regions is always a subset of their joint distribution. Moreover, it is possible to construct joint distributions for which each individual decision region outside the convex hull becomes inaccessible, even if there may not exist one unique joint distribution at which all synergetic regions are lost simultaneously. This volatility due to the dependence between variables appears suitable for a notion of synergy. Similarly, the resulting unique information appears suitable since it ensures that a variable with unique information must provide access to some additional decision region. Finally, the obtained unique and redundant information is sensible [9] since it only depends on the marginal distributions with the target. The operational interpretation can be strengthened further such that the implication between accessible regions and partial information holds in both directions by revising Lemmas A1 and A2 with a strictly convex generator function.

We perform the decomposition on a pointwise lattice using the Blackwell join since it is possible to represent f-information as the expected value of quantifying the Neyman–Pearson region boundary (zonogon perimeter) for indicator variables (pointwise channels). Since the pointwise measures satisfy a triangle inequality, we mentioned the oversimplified intuition of pointwise f-information as the length of the zonogon perimeter. Correspondingly, if we identified an information measure that behaved more like the area of the zonogon (which could also maintain their ordering), then we would need to decompose it on a pointwise lattice using the Blackwell meet to achieve non-negativity. We assume that most information measures behave more similar to quantifying the boundary length rather than its area, since the boundary segments can directly be obtained from the conditional probability distribution and do not require an actual construction from the likelihood-ratio test.

In the literature, PIDs have been defined based on different ordering relations [16], the Blackwell order being only one of them. We think that this diversity is desirable since each approach provides a different operational interpretation of redundancy and synergy. For this reason, we wonder if obtaining a non-negative decomposition with the inclusion–exclusion relation for other ordering relations was possible when transferring them to a pointwise perspective or from mutual information to other information measures.

Studying the relations between different information measures for the same decomposition method may provide further insights into their properties, as demonstrated by the example of total variation in Section 4.2. The ability to decompose different information measures is also a necessity to apply the method in a variety of areas, since each information measure can then provide the operational meaning within its respective domains. To ensure consistency between related information measures, we allowed the re-definition of information addition, as demonstrated in the example of Rényi-information in Section 3.6, which also opens new possibilities for satisfying the inclusion–exclusion relation.

There is currently no universally accepted definition of conditional Rényi information. Assuming that $I_{{\mathrm{R}}_a}(T;{\mathbf{S}}_i\mid {\mathbf{S}}_j)$ should capture the information that ${\mathbf{S}}_i$ provides about T when already knowing the information from ${\mathbf{S}}_j$, then one could propose that this quantity should correspond to the according partial information contributions (unique/synergetic) and, thus, the definition of Equation (53).

With this in mind, it is also possible to define, model, decompose, and trace Transfer Entropy [33], used in the analysis of complex systems, for each presented information measure with the methodology of Section 4.2.

$$I_{{\mathrm{R}}_a}(T;{\mathbf{S}}_i\mid {\mathbf{S}}_j)I_{{\mathrm{R}}_a}(T;{\mathbf{S}}_i,{\mathbf{S}}_j)\ominus I_{{\mathrm{R}}_a}(T;{\mathbf{S}}_j)$$

(53)

Finally, studying the corresponding definitions for continuous random variables and identifying suitable information measures for specific applications would be interesting directions for future work.

6. Conclusions

In this work, we demonstrated a non-negative PID in the framework of Williams and Beer [1] for any f-information with practical operational interpretation and the conversion of measures between decomposition lattices. We demonstrated that the decomposition of f-information can be used to obtain a non-negative decomposition of Rényi-information, for which we re-defined the addition to demonstrate that its results satisfy an inclusion–exclusion relation. Finally, we demonstrated how the proposed decomposition method can be used for tracing the flow of information through Markov chains and how the decomposition obtains different properties depending on the chosen information measure.