Causally Informative Entropic Inequalities within Families of Distributions with Shared Marginals

Abstract

The joint probability distribution of observable variables from a system is constrained by the underlying causal structure. In the presence of hidden variables, untestable independencies that involve hidden variables lead to testable causally-imposed inequality constraints for observable variables, whose violation can reject the compatibility of a causal structure with data. One type of causally informative inequalities is entropic inequalities, which appear in the space of entropic terms associated with the distribution of observable variables. We derive a new type of minimum information (minInf) entropic inequalities that substantially increases causal inference power. These new entropic inequalities appear when considering the constraints that the causal structure imposes on entropic terms determined by information minimization within families of distributions that preserve sets of marginals shared with the original distribution. We introduce a new family of minInf data processing inequalities and a procedure to recursively combine different types of data processing inequalities to create tighter testable entropic inequalities. We extensively illustrate the applicability of this procedure in the instrumental causal scenario, integrating the new inequalities with standard instrumental entropic inequalities constructed with multivariate instrumental sets. We also provide additional examples with other types of entropic inequalities, such as the Information Causality and Groups-Decomposition inequalities.

1. Introduction

Understanding which causal structures are compatible with a set of observational data is a common question in science. The underlying causal structure of a system creates constraints on the probability distribution of variables generated from it [1,2,3], which helps to reversely infer which causal structures are compatible with the data. Causal learning algorithms based on conditional independencies [1,2,4] reconstruct a partially oriented graph [5] that represents the equivalence class of all causal structures compatible with the set of conditional independencies present in the distribution of the observable variables (the so-called Markov equivalence class). However, in most real-world scenarios, the components of a system are only partially observed, and the presence of hidden variables creates dependencies among the observable variables that limit the degree to which Markov equivalence classes narrow down the set of causal structures compatible with the data.

Beyond statistical independencies in the joint distribution, the underlying causal structure can also be reflected in other equality constraints imposed to the observable variables. These constraints comprise functional equality constraints [6,7] and independencies that originate from further assumptions about the functional form of the generative mechanisms of the variables [3,8,9,10,11,12,13]. Additionally, nonverifiable conditional independencies that involve hidden variables can manifest themselves through inequality constraints that only involve observable variables [14,15,16]. Unlike equality constraints, inequality constraints provide necessary but not sufficient conditions for the compatibility of data with a certain causal structure. Data violations of inequalities enforced by a causal structure allow discarding that causal structure as the one generating the data. Accordingly, causal inference power is increased when deriving tighter inequalities. Causally informative inequalities comprise inequalities derived in the probability space, such as Bell-type inequalities [17,18], instrumental inequalities [19,20], and interventional inequalities [21], as well as entropic inequalities derived in the space of the entropic terms associated with the observable variables [14,22].

Causally informative entropic inequalities can be derived with two alternative approaches. One approach is to derive a specific entropic inequality departing from a concrete equality that involves hidden variables and then using the conditional independencies associated with the causal structure to derive for the two sides of the equality upper and lower bounds which do not contain hidden variables [23,24,25,26]. In order to derive a testable inequality that only contains observable variables, this approach relies on the data processing (DP) inequality [27] to replace hidden variables by less informative observable variables. In a second approach, all the testable causally informative inequalities imposed by a causal structure are derived reducing the set of equalities and inequalities that characterize the whole system –comprising observable and hidden variables– to the marginal scenario associated with the observable variables [14,24]. This approach combines the inequalities that define the Shannon entropic cone [28], i.e., associated with the nonnegativity, monotonicity, and submodularity properties of entropy, and all additional independence constraints related to all the variables in the causal structure. Subsequently, variable elimination is performed to extract the resulting constraints that only involve observable variables. While this marginalization problem is algorithmically solvable [14], its implementation for large systems is challenging and furthermore does not provide an explicit constructive recipe that allows tracing the resulting inequalities in terms of the existing conditional independencies.

These methods to construct causally informative entropic inequalities traditionally rely on entropic terms associated with the original joint distribution of the observable variables. However, specifically for the so-called Groups-Decomposition inequalities [25,26], it has been shown [26] that new as well as tighter more informative inequalities can be derived if the entropic terms of the original joint distribution are combined with maximum entropy entropic terms. Chicharro and Nguyen [26] introduced a DP inequality for the maximum entropy measure of unique information [29], a measure originally proposed to decompose mutual information into redundant, unique, and synergistic components [30]. The maximum entropy unique information is defined by an information minimization within a family of distributions constrained to share some marginals of the original distribution of the observable variables. In this work, we capitalize on an extended combination of the original entropic terms and additional entropic terms defined by information minimization within a broader set of different families. We introduce novel procedures to obtain new and tighter entropic inequalities incorporating these additional entropic terms.

To motivate our derivations, we first proceed with the first approach to derive causally informative entropic inequalities. That is, we focus on concrete causal structures and apply an explicit procedure that involves DP inequalities. Specifically, we focus on instrumental entropic inequalities [24], which appear in the causal scenario of instrumental variables [19]. We derive new instrumental entropic inequalities comprising maximum entropy unique information terms. This allows us to characterize a procedure to recursively combine different DP inequalities to create tighter instrumental entropic inequalities. We then introduce a much wider family of DP inequalities for information terms determined by constrained minimization. These minimum information (minInf) terms are defined within families of distributions that share sets of marginals of the original joint distribution. MinInf DP inequalities are then used to derive tighter instrumental inequalities. Subsequently, we indicate how new entropic inequalities can be derived not only from the sequential application of new minInf DP inequalities, but also as a marginalization problem.

Overall, the minInf DP inequalities that we develop, and the procedure to sequentially combine them, provide a general tool to derive new types of entropic inequalities and to extend existing ones thanks to the incorporation of additional information terms to obtain tighter lower bounds. To illustrate the generality of these tools, we finally also examine how other well-known types of causally informative entropic inequalities [23,25,26] can equally be extended into inequalities with an increased causal inference power. While entropic inequalities have also been formulated for quantum systems [23,31], in this work we restrict our derivation to classical Shannon entropy measures. The Discussion section comments on potential extensions.

This paper is organized as follows. In Section 2, we review existing results relevant for our work. In Section 3.1, we derive instrumental entropic inequalities with unique information terms. In Section 3.2, we compare the new inequalities to standard instrumental inequalities with multivariate instrumental sets, identifying conditions in which the new inequalities provide additional causal inference power. In Section 3.3, we show that causal inference power is increased not only using the DP inequality of unique information instead of the standard DP inequality, but also iteratively combining them. In this way, we identify a procedure to iteratively combine multiple DP inequalities. This procedure is further developed in Section 3.4, where we introduce a general type of DP inequalities for minInf information terms and combine them recursively to add observable minInf information terms as lower bounds of information terms with hidden variables. In Section 3.5, we apply this procedure specifically to build more causally informative instrumental entropic inequalities. Section 3.6 reframes the use of minInf information terms for causal learning with the optics of a marginalization problem. We indicate how to extend the Shannon entropy cone to minInf Shannon entropy cones that jointly characterize minInf families. Marginalization of the hidden variables within this joint space produces also the causally informative entropic inequalities that contain minInf entropic terms. Finally, to provide broader examples of applicability of our methods, in Section 3.7 we show how to extend two other types of causally informative entropic inequalities, namely Groups-Decompositions inequalities [25,26] and the Information Causality inequality [23,31].

2. Methods

In this section we review the relation between causal graphs and conditional independencies, the standard data processing inequality, the standard instrumental entropic inequality, as well as the formulation of minimum mutual information quantities, comprising a measure of maximum entropy unique information.

2.1. Causal Graphs and Conditional Independencies

We review Directed Acyclic Graphs (DAGs) and the relation between causal structures and dependencies. A DAG $G=(\overline{\mathbf{V}};\mathcal{E})$ associated with a set of random variables $\overline{\mathbf{V}}=\{{\overline{V}}_1,\dots ,{\overline{V}}_n\}$ consists of nodes $\overline{\mathbf{V}}$ and edges $\mathcal{E}$ between the nodes, where $\overline{V}$ refers both to a variable and its corresponding node. Note that in general $\overline{\mathbf{V}}$ can comprise both observable and hidden variables; we will later use specifically letter U for hidden variables. The set of edges $\mathcal{E}$ contains $({\overline{V}}_i;{\overline{V}}_j)\in \mathcal{E}$ for each arrow ${\overline{V}}_i\to {\overline{V}}_j$, which indicates a causal connection in the system generating the variables. The structure of edges in the graph removing arrowheads is called the skeleton of the graph. The graph is acyclic because we consider causal mechanisms not to be instantaneous and any causal cycle spans in time.

A path in G is a sequence of (at least two) distinct nodes ${\overline{V}}_1,\dots ,{\overline{V}}_m,$ such that there is an edge between ${\overline{V}}_k$ and ${\overline{V}}_{k+1}$ for all $k=1,\dots ,m-1$. If all edges are directed as ${\overline{V}}_k\to {\overline{V}}_{k+1}$ the path is a causal or directed path. A node ${\overline{V}}_i$ is a collider in a path if it has incoming arrows ${\overline{V}}_{i-1}\to {\overline{V}}_i\leftarrow {\overline{V}}_{i+1}$ and is a noncollider otherwise. If there is an arrow ${\overline{V}}_i\to {\overline{V}}_j$, then ${\overline{V}}_i$ is a parent of ${\overline{V}}_j$, and ${\overline{V}}_j$ is a child of ${\overline{V}}_i$. A node ${\overline{V}}_i$ is called an ancestor of ${\overline{V}}_j$ if there is a directed path from ${\overline{V}}_i$ to ${\overline{V}}_j$. Conversely, in this case ${\overline{V}}_j$ is a descendant of ${\overline{V}}_i$. We use bidirected arcs ${\overline{V}}_i\leftrightarrow {\overline{V}}_j$ to indicate the presence of a nondirected path between ${\overline{V}}_i$ and ${\overline{V}}_j$ consisting only of hidden noncolliders.

A causal graph accurately represents the generative mechanisms of a system when a variable ${\overline{V}}_i$ is a parent of another variable ${\overline{V}}_j$ if and only if it is an argument of an underlying functional equation that captures the mechanisms that generate ${\overline{V}}_j$. This creates a relation between the conditional independencies that hold between variables in the system and a graphical criterion of separability between the nodes, called d-separation [32]. The criterion of d-separation states that two nodes X and Y are d-separated given a set of nodes $\mathbf{S}$ if and only if no $\mathbf{S}$-active paths exist between X and Y. A path is active given the conditioning set $\mathbf{S}$ ($\mathbf{S}$-active) if no noncollider in the path belongs to $\mathbf{S}$ and every collider in the path either is in $\mathbf{S}$ or has a descendant in $\mathbf{S}$. A causal structure G and a generated probability distribution $p\left(\overline{\mathbf{V}}\right)$ are faithful [1,2] to one another when a conditional independence between X and Y given $\mathbf{S}$ –denoted by $X{\perp }_{P}Y|\mathbf{S}$– holds if and only if X and Y are d-separated given $\mathbf{S}$ –denoted by $X{\perp }_{G}Y|\mathbf{S}$.

The inference of the causal structure of a system from data generated from the system relies on this link between the causal structure and independencies. Causal learning algorithms that use conditional independencies to reconstruct a partially oriented graph [1,2,4] rely on the assumption of faithfulness in order to determine the skeleton of the graph and to apply rules of orientation of the edges. On the other hand, in the case that causally informative inequalities are used to rule out causal structures [14,24,26], it is only required to assume the substantially weaker assumption that d-separability implies conditional independence. Under this assumption, if a causal structure implies the set of independencies that lead to the fulfillment of the inequality, its violation allows discarding that causal structure. The assumption of faithfulness is not required because if unfaithful independencies are present in the data, which do not follow from the causal structure, this may decrease the power to reject causal structures, but does not lead to incorrect rejections.

Note that the assumption that graphical separability implies statistical conditional independence is substantially weaker than the converse assumption that statistical conditional independence implies graphical separability. A counterexample of the latter is the X-OR logical gate. On the other hand, if the causal graph reflects the underlying structure of mechanisms involved in generating the variables, all statistical dependencies need to originate from some paths of influence between the variables. If some variables are conditionally dependent while the graph indicates that they are d-separated, then the graph must be misrepresenting the paths that create the observed dependence.

2.2. The Data Processing Inequality

The data processing inequality (DP inequality) of mutual information indicates that information cannot be increased in a Markov chain [27].

Lemma 1  

(Data processing inequality of conditional mutual information). Let $\overline{\mathbf{Z}}$, $\mathbf{D}$, ${\mathbf{D}}^{\prime }$, and $\mathbf{E}$ be four nonoverlapping sets of variables. If $\overline{\mathbf{Z}}\perp {\mathbf{D}}^{\prime }|\mathbf{D}\mathbf{E}$, then it follows that $I(\overline{\mathbf{Z}};\mathbf{D},{\mathbf{D}}^{\prime }|\mathbf{E})=I(\overline{\mathbf{Z}};\mathbf{D}|\mathbf{E})\ge I(\overline{\mathbf{Z}};{\mathbf{D}}^{\prime }|\mathbf{E})$.

While the DP inequality is often used only to refer to the inequality between the information carried by $\mathbf{D}$ and ${\mathbf{D}}^{\prime }$, we will also apply the equality of the information carried by $\{\mathbf{D},{\mathbf{D}}^{\prime }\}$ and $\mathbf{D}$ alone.

2.3. The Instrumental Entropic Inequality

We here revise the instrumental entropic inequality [24]. We provide its full derivation because this helps to identify how new entropic inequalities can be derived. Consider the causal structures of Figure 1A, with all variables observable except U hidden. We use a notation of the variables consistent with the role of Z, X, Y, and U in the work that introduced the instrumental inequality [19] and its entropic formulation [24]. The diagram represents several causal structures, depending on how the dashed edges are instantiated (or removed) as additional causal connections. These additional connections are constrained by the acyclic nature of the causal graph, for example $Z\to Y$ or $Z\leftrightarrow Y$ are valid, while $Z\leftarrow Y$ is not, since it would lead to the existence of a cycle. For all the causal structures of Figure 1A, no conditional independencies between variables in $\{X,Y,Z\}$ exist that involve conditioning only on observable variables. Accordingly, the reconstruction of the causal structure based on conditional independencies, for example using the PC algorithm of Spirtes et al. [1], results in all cases in a reconstructed graph in which nodes $\{X,Y,Z\}$ are all connected. In particular, also for the causal structure in which the dashed edges are removed, a reconstructed edge $Z-Y$ is obtained, even if not present in the actual skeleton of that causal structure. This is due to the fact that blocking the path $Z\to X\to Y$ by conditioning on X activates the path $Z\to X\leftarrow U\to Y$.

The instrumental entropic inequality provides a causally informative test to reject the compatibility of a data set with this causal structure in which the dashed edges are not present. Even if no independencies between $\{Z,X,Y\}$ exist in the marginal scenario in which U is hidden, this causal structure contains untestable independencies that involve the hidden variable U, comprising $Z\perp U|W$ and $Z\perp Y|UXW$. These untestable independencies impose additional constraints that manifest themselves in an inequality between information terms, namely the instrumental entropic inequality. For later convenience, we formulate the standard instrumental entropic inequality allowing for a multivariate $\mathbf{Z}$:

Proposition 1  

(Instrumental entropic inequality). Consider the variables $\mathbf{Z}$, X, Y, ${\mathbf{B}}_0$, and U, all observable except U a hidden variable. Consider that the causal structure is such that, for all $Z_i\in \mathbf{Z}$, no pair from $\{Z_i,X,Y\}$ is separable given that U is hidden. Consider that the causal structure imposes the existence of the nontestable independencies $\mathbf{Z}\perp U|{\mathbf{B}}_0$ and $\mathbf{Z}\perp Y|UX{\mathbf{B}}_0$. These independencies result in the testable inequality

$$H\left(X\right|{\mathbf{B}}_0)\ge I(\mathbf{Z};X|{\mathbf{B}}_0)+I(\mathbf{Z};Y|{\mathbf{B}}_0,X).$$

(1)

Proof.  

The mutual information $I(\mathbf{Z};U,X|{\mathbf{B}}_0)$ can be decomposed applying the chain rule in two alternative orders. If information with U is considered first

$$\begin{array}{c}\hfill I(\mathbf{Z};U,X|{\mathbf{B}}_0)\stackrel{\left(a\right)}{=}I(\mathbf{Z};U|{\mathbf{B}}_0)+I(\mathbf{Z};X|{\mathbf{B}}_0,U)\stackrel{\left(b\right)}{\le }H\left(X\right|{\mathbf{B}}_0).\end{array}$$

(2)

Equality $\left(a\right)$ applies the chain rule of mutual information. Inequality $\left(b\right)$ holds because $I(\mathbf{Z};U|{\mathbf{B}}_0)=0$, given the independence $\mathbf{Z}\perp U|{\mathbf{B}}_0$, and by definition $I(\mathbf{Z};X|{\mathbf{B}}_0,U)$ is smaller than or equal to $H\left(X\right|{\mathbf{B}}_0,U)$, which by monotonicity of entropy under conditioning is smaller than or equal to $H\left(X\right|{\mathbf{B}}_0)$. Considering now the chain rule with X first,

$$\begin{array}{c}\hfill I(\mathbf{Z};U,X|{\mathbf{B}}_0)\stackrel{\left(a\right)}{=}I(\mathbf{Z};X|{\mathbf{B}}_0)+I(\mathbf{Z};U|{\mathbf{B}}_0,X)\stackrel{\left(b\right)}{\ge }I(\mathbf{Z};X|{\mathbf{B}}_0)+I(\mathbf{Z};Y|{\mathbf{B}}_0,X).\end{array}$$

(3)

Equality $\left(a\right)$ applies the chain rule of mutual information. Inequality $\left(b\right)$ holds because $\mathbf{Z}\perp Y|UX{\mathbf{B}}_0$ implies the DP inequality $I(\mathbf{Z};U|{\mathbf{B}}_0,X)\ge I(\mathbf{Z};Y|{\mathbf{B}}_0,X)$. Combining the upper bound $H\left(X\right|{\mathbf{B}}_0)$ and the lower bound $I(\mathbf{Z};X|{\mathbf{B}}_0)+I(\mathbf{Z};Y|{\mathbf{B}}_0,X)$ proves the testable inequality. □

The instrumental entropic inequality holds in Figure 1A with, $\mathbf{Z}=Z$ and ${\mathbf{B}}_0=W$. It equally holds with changes in $Z-W-Y$, as long as W is a noncollider. Note that while the instrumental inequality gets its name from the possibility to use Z as a causal intervention instrument that can be manipulated (intervened) to estimate the causal effect that X has on Y [20,33,34] in fact the inequality is equally fulfilled with $Z\leftrightarrow X$, since this does not alter the independence $Z\perp U|W$. That is, the instrumental inequality also holds for causal structures where Z is not a causal intervention instrument. Since in this work we study causal structure learning and not the identification of causal effects, we will refer to a set of variables as a causal discovery instrumental set purely based on the fulfillment of the independence $\mathbf{Z}\perp U|{\mathbf{B}}_0$. The reason not to involve the independence $\mathbf{Z}\perp Y|UX{\mathbf{B}}_0$ in this criterion will become clear in Section 3.2. Furthermore, since there is no possible confusion within this work, we will abbreviate causal discovery instrumental set simply as instrumental set.

Proposition 1 states a straightforward extended version of the basic instrumental entropic inequality in the sense that it comprises a multivariate $\mathbf{Z}$. We will refer to this inequality as the standard instrumental entropic inequality. This multivariate version will be needed for comparison with the new types of instrumental entropic inequalities we will introduce. Note that we purposely have excluded further straightforward generalizations, such as a multivariate $\mathbf{X}$, $\mathbf{Y}$, and $\mathbf{U}$. We will add more generalizations in the Results section, but this simple version is suited to identify the key components of the inequality and its relation to the causal structure, as will be examined in Section 3.2.

Importantly, a causal structure that fulfills the nontestable independencies $\mathbf{Z}\perp U|{\mathbf{B}}_0$ and $\mathbf{Z}\perp Y|UX{\mathbf{B}}_0$ imposes the fulfillment of the inequality of Proposition 1 to any data set generated from that causal structure. In that case, we will say that the inequality is causally fulfilled. On the other hand, for a data set generated with another causal structure, the entropic inequality may equally be fulfilled, even if its fulfillment was not imposed by the causal structure. In that case, we will say that the inequality is statistically fulfilled. The causal inference power of an inequality emanates from the possibility to discard a causal structure that imposes the causal-fulfillment of the inequality when the violation of the inequality is verified from data. As mentioned in Section 2.1, in order to be able to reject causal structures based on the violation of causally informative inequalities, we will work under the assumption that causal separability (d-separation) implies statistical conditional independence. That is, we assume that for a causal structure that causally imposes an inequality, the inequality indeed is fulfilled because the causal structure creates the independencies that lead to the inequality.

2.4. Constrained Minimum Mutual Information and Maximum Entropy Unique Information

Problems of constrained optimization of information-theoretic quantities within families that share marginal distributions often appear in the study of communication channels [35]. Furthermore, minimum information methods have been proposed for machine learning and signal processing tasks [36,37] as a generalization of the maximum entropy principle [38]. As developed in the Results below, our extension of causally informative entropic inequalities relies on the use of minimum information (minInf) terms that are defined within families of distributions that share sets of marginals with the original joint distribution.

In general, minimization constraints can comprise both inequality and equality equations that are imposed to joint distributions of variables. A set of constraints determines a family $\Delta P$ of probability distributions within which the mutual information term of interest is to be minimized, namely among all distributions compatible with the fulfillment of the constraints. In this work, we focus on minInf terms defined within families of distributions that preserve sets of marginals of the joint distribution associated with a data set. In general, given a joint distribution $P\left(\overline{\mathbf{V}}\right)$ for $\overline{\mathbf{V}}$ variables, a minInf term is defined as

$$\underset{Q\in \Delta P}{min}{I}_Q({\overline{\mathbf{V}}}_1;{\overline{\mathbf{V}}}_2|{\overline{\mathbf{V}}}_3),$$

(4)

where ${\overline{\mathbf{V}}}_i$, $i=1,\dots ,3$ are subsets of $\overline{\mathbf{V}}$ and $\Delta P$ is the family of distributions determined by the set of constraints imposed to the distributions. Accordingly, the minimization within the family corresponds to a constrained minimization subject to the constraints that define the family. When the constraints impose the preservation of a set of marginal distributions of the original $P\left(\overline{\mathbf{V}}\right)$, that is, when $Q\in \Delta P$ is subject to $Q\left({\overline{\mathbf{V}}}_S\right)=P\left({\overline{\mathbf{V}}}_S\right)$ for a certain number of subsets S of $\overline{\mathbf{V}}$, then the constraints constitute a set of affine linear equality constraints on the joint distribution $Q\left(\overline{\mathbf{V}}\right)$.

In this work we prove some general properties of minInf information terms that render them useful for the derivation of more powerful causally informative testable entropic inequalities. These properties include a data processing inequality for minInf terms (Proposition 5) and a procedure to iteratively combine minInf data processing inequalities (Theorem 1). We then use these properties for the construction of some specific entropic inequalities, such as extended instrumental entropic inequalities. To the best of our knowledge, although the use of minimum information quantities appears in the study of communication channels and has been formulated in machine learning problems as mentioned above, the properties we introduce have not been derived before, and the use of minInf terms to derive causally informative entropic inequalities is new.

As a precedent to this work, Chicharro and Nguyen [26] showed how to apply to causal structure learning a measure of maximum entropy unique information previously introduced in [29]. Maximum entropy measures correspond to a specific subcase of minInf terms of the form of Equation (4). Concretely, when $P({\overline{\mathbf{V}}}_1,{\overline{\mathbf{V}}}_3)$ is among the preserved marginals in $\Delta P$, then the entropy $H_Q\left({\overline{\mathbf{V}}}_1\right|{\overline{\mathbf{V}}}_3)$ is fixed, and the minimization of $I_Q({\overline{\mathbf{V}}}_1;{\overline{\mathbf{V}}}_2|{\overline{\mathbf{V}}}_3)$ is equivalent to the maximization of $H_Q\left({\overline{\mathbf{V}}}_1\right|{\overline{\mathbf{V}}}_2,{\overline{\mathbf{V}}}_3)$.

We now revise the definition of the maximum entropy unique information [29] and the relevant properties derived in Chicharro and Nguyen [26]. This will serve as reference to generalize more general data processing inequalities for minInf terms. The concept of unique information was originally introduced [30] as part of a nonnegative decomposition of the joint mutual information that a set of predictor variables has about a target variable $\overline{\mathbf{Z}}$. In the simplest scenario with two (possibly multivariate) predictors $\{{\mathbf{D}}_1,{\mathbf{D}}_2\}$, the unique information of predictor ${\mathbf{D}}_i$ with respect to the reference predictor ${\mathbf{D}}_j$ quantifies the exclusive information about $\overline{\mathbf{Z}}$ obtained from ${\mathbf{D}}_i$ and not from ${\mathbf{D}}_j$. The other components of the decomposition quantify redundant and synergistic information terms. Alternative formulations of this decomposition of mutual information have been introduced, e.g., [29,39,40,41], using definitions alternative to the maximum entropy formulation of unique information of [29]. However, the application of the maximum entropy unique information measure to causal inference [26] does not rely on its embedding within the framework of [30], but only on certain properties that render it useful to derive testable causally informative information inequalities. We here revise its definition and these properties. In general, we will use unique information to refer concretely to the maximum entropy unique information.

While the measure was originally introduced in the bivariate unconditional case [29], we here revise the conditional unique information measure as presented in [26]. For sets of variables $\overline{\mathbf{Z}}$, ${\mathbf{D}}_1$, ${\mathbf{D}}_2$, and ${\mathbf{O}}_1$, the unique information of predictor ${\mathbf{D}}_1$ with respect to the reference ${\mathbf{D}}_2$ about the target $\overline{\mathbf{Z}}$, conditioning on ${\mathbf{O}}_1$, is defined as

$$I(\overline{\mathbf{Z}};{\mathbf{D}}_1\backslash \backslash {\mathbf{D}}_2\left|{\mathbf{O}}_1\right)\equiv \underset{Q\in \Delta P}{min}{I}_Q(\overline{\mathbf{Z}};{\mathbf{D}}_1|\mathbf{E}),$$

(5)

where $\mathbf{E}\equiv \{{\mathbf{D}}_2,{\mathbf{O}}_1\}$, and $\Delta P$ is the family of distributions on $\{\overline{\mathbf{Z}},{\mathbf{D}}_1,{\mathbf{D}}_2,{\mathbf{O}}_1\}$ that preserve the marginals $P(\overline{\mathbf{Z}},{\mathbf{D}}_1,{\mathbf{O}}_1)$ and $P(\overline{\mathbf{Z}},{\mathbf{D}}_2,{\mathbf{O}}_1)$ of the original $P(\overline{\mathbf{Z}},{\mathbf{D}}_1,{\mathbf{D}}_2,{\mathbf{O}}_1)$. The notation $I_Q$ is used to indicate that the mutual information is calculated on the probability distribution Q. We use $I(\overline{\mathbf{Z}};{\mathbf{D}}_1\backslash \backslash {\mathbf{D}}_2|{\mathbf{O}}_1)$ to refer to the unique information of ${\mathbf{D}}_1$ with reference ${\mathbf{D}}_2$, conditioning on ${\mathbf{O}}_1$, compared to $I(\overline{\mathbf{Z}};{\mathbf{D}}_1|{\mathbf{D}}_2,{\mathbf{O}}_1)$, which is the standard conditional information of ${\mathbf{D}}_1$ conditioning on $\{{\mathbf{D}}_2,{\mathbf{O}}_1\}$. Note that the constraints on $\Delta P$ are such that they divide the conditioning set $\mathbf{E}$ into the variables ${\mathbf{O}}_1$ included in $P(\overline{\mathbf{Z}},{\mathbf{D}}_1,{\mathbf{O}}_1)$ and the variables ${\mathbf{D}}_2$, that appear only in the marginal $P(\overline{\mathbf{Z}},{\mathbf{D}}_2,{\mathbf{O}}_1)$, where ${\mathbf{D}}_1$ is excluded. We enumerate ${\mathbf{O}}_1$ with subindex 1 because when dealing with more general minInf terms in Section 3.4 we will distinguish multiple subsets of $\mathbf{E}$ preserved in different marginal distributions. We use the notation ${\mathbf{D}}_1\backslash \backslash {\mathbf{D}}_2$ instead of the notation ${\mathbf{D}}_1\backslash {\mathbf{D}}_2$ originally introduced by [29] to differentiate it from the set notation ${\mathbf{D}}_1\backslash {\mathbf{D}}_2$, which indicates the subset of variables in ${\mathbf{D}}_1$ that is not contained in ${\mathbf{D}}_2$. The measure defined in Equation (5) is equivalently a maximum entropy measure because the information minimization can equally be formulated as an entropy maximization, since all distributions within $\Delta P$ preserve the conditional entropy $H\left(\overline{\mathbf{Z}}\right|{\mathbf{D}}_2,{\mathbf{O}}_1)$.

Several properties are important for its use into causally informative inequalities. First, by construction [29], the conditional unique information is bounded as

$$min\{I(\overline{\mathbf{Z}};{\mathbf{D}}_1|{\mathbf{O}}_1),I(\overline{\mathbf{Z}};{\mathbf{D}}_1|{\mathbf{D}}_2,{\mathbf{O}}_1)\}\ge I(\overline{\mathbf{Z}};{\mathbf{D}}_1\backslash \backslash {\mathbf{D}}_2\left|{\mathbf{O}}_1\right)\ge 0.$$

(6)

Second, the unique information is monotonic in the predictor argument:

Lemma 2  

(Monotonicity of unique information in the predictor argument). The maximum entropy conditional unique information is monotonic on its second argument, corresponding to the non-referent predictor:

$$I(\overline{\mathbf{Z}};\mathbf{D},{\mathbf{D}}^{\prime }\backslash \backslash {\mathbf{D}}_{\mathbf{2}}|{\mathbf{O}}_1)\ge I(\overline{\mathbf{Z}};\mathbf{D}\backslash \backslash {\mathbf{D}}_{\mathbf{2}}\left|{\mathbf{O}}_1\right).$$

(7)

In relation to Equation (5), ${\mathbf{D}}_1=\{\mathbf{D},{\mathbf{D}}^{\prime }\}$. This property was derived in Lemma 3 of [42] for the unconditional case and extended to the conditional case in Lemma 2 of [26]. The proof is provided in Appendix A.

Third, a DP inequality was derived in Chicharro and Nguyen [26] for the maximum entropy unique information:

Lemma 3  

(Conditional unique information data processing inequality). Let $\overline{\mathbf{Z}}$, $\mathbf{D}$, ${\mathbf{D}}^{\prime }$, ${\mathbf{D}}_{\mathbf{2}}$, and ${\mathbf{O}}_1$ be five nonoverlapping sets of variables. If $I(\overline{\mathbf{Z}};{\mathbf{D}}^{\prime }|\mathbf{D},{\mathbf{O}}_1)=0$, then $I(\overline{\mathbf{Z}};\mathbf{D},{\mathbf{D}}^{\prime }\backslash \backslash {\mathbf{D}}_{\mathbf{2}}|{\mathbf{O}}_1)=I(\overline{\mathbf{Z}};\mathbf{D}\backslash \backslash {\mathbf{D}}_{\mathbf{2}}|{\mathbf{O}}_1)\ge I(\overline{\mathbf{Z}};{\mathbf{D}}^{\prime }\backslash \backslash {\mathbf{D}}_{\mathbf{2}}\left|{\mathbf{O}}_1\right)$.

Again, the proof is provided in Appendix A to serve as guidance for posterior extensions. This DP inequality is analogous to the standard DP inequality of the mutual information (Lemma 1), except that it requires the independence $\overline{\mathbf{Z}}\perp {\mathbf{D}}^{\prime }|\mathbf{D}{\mathbf{O}}_1$ instead of $\overline{\mathbf{Z}}\perp {\mathbf{D}}^{\prime }|\mathbf{D}\mathbf{E}$, with $\mathbf{E}=\{{\mathbf{D}}_{\mathbf{2}},{\mathbf{O}}_1\}$. Note that all the variables involved in the independence $\overline{\mathbf{Z}}\perp {\mathbf{D}}^{\prime }|\mathbf{D}{\mathbf{O}}_1$ are included in the marginal $P(\overline{\mathbf{Z}},{\mathbf{D}}^{\prime },\mathbf{D},{\mathbf{O}}_1)$ preserved in the family $\Delta P$ that defines $I(\overline{\mathbf{Z}};\mathbf{D},{\mathbf{D}}^{\prime }\backslash \backslash {\mathbf{D}}_{\mathbf{2}}|{\mathbf{O}}_1)$.

3. Results

To develop how minInf terms can be used in causally informative entropic inequalities, we start from the scenario of the standard instrumental entropic inequality (Figure 1A) and consider changes in the causal structure. We first derive (Section 3.1) an instrumental entropic inequality that applies the DP inequality of unique information. We then address the embedding of this new type of inequalities together with standard instrumental inequalities derived with multivariate instrumental sets and we illustrate that they can provide additional causal inference power (Section 3.2). In Section 3.3, we examine instrumental entropic inequalities in which the DP inequality of conditional mutual information and unique information are combined. This analysis reveals how different types of DP inequalities can recursively be applied. In Section 3.4, we introduce a type of DP inequalities for minInf terms, which encompasses as subcases the DP inequalities of conditional mutual information and unique information. We show how to recursively apply these DP inequalities to obtain sums of observable information terms as lower bounds of unobservable information terms. In Section 3.5, we apply this procedure to construct more powerful instrumental entropic inequalities. In Section 3.6, we reexamine more broadly the derived minInf inequalities from a geometrical perspective, in connection with Shannon entropy cones. Finally, in Section 3.7, we apply the procedures developed in Section 3.4 to other types of entropic inequalities beyond the instrumental inequality scenario.

3.1. Instrumental Entropic Inequalities with Maximum Entropy Unique Information Terms: The Case with One Data Processing Inequality Applied

We start considering how to construct instrumental entropic inequalities with the causal structures of Figure 1B. Again, the graph displays several causal structures depending on the instantiation of the dashed edges. Similar to the case of Figure 1A, a requirement for any instrumental entropic inequalities to be causally fulfilled is that the dashed edges between Z and Y as well as between Z and U are removed. Therefore, we focus on this case with no edges between Z and Y and between Z and U. A difference between Figure 1A,B is that W is a noncollider in Figure 1A, leading to $Z\cancel{\perp }Y|UX$ and $Z\perp Y|UXW$ while W is a collider in Figure 1B, such that $Z\perp Y|UX$ and $Z\cancel{\perp }Y|UXW$. For Figure 1B, the instrumental inequality of Proposition 1 can be applied with $\mathbf{Z}=Z$ and ${\mathbf{B}}_0=\varnothing$. The required independencies $\mathbf{Z}\perp U|{\mathbf{B}}_0$ and $\mathbf{Z}\perp Y|UX{\mathbf{B}}_0$ are fulfilled, namely they correspond to $Z\perp U$ and $Z\perp Y|UX$. This leads to

$$H\left(X\right)\ge I(Z;X)+I(Z;Y|X).$$

(8)

On the contrary, the fact that W is a collider leads to a dependence $Z\cancel{\perp }Y|UXW$, and hence in Figure 1B Proposition 1 cannot be applied selecting ${\mathbf{B}}_0=W$. Note that being able to condition on ${\mathbf{B}}_0=W$ would be advantageous because, following the derivation of Proposition 1, it would lead to a tighter upper bound $H\left(X\right|W)$ instead of $H\left(X\right)$. Since what prevents deriving an instrumental entropic inequality with ${\mathbf{B}}_0=W$ is that $Z\cancel{\perp }Y|UXW$, as opposed to $Z\perp Y|UX$, we can consider if using the unique information DP inequality is useful in this case. This is because the unique information has a DP inequality (Lemma 3) that differs from the one of conditional mutual information (Lemma 1) in that it is associated with a conditional independence that excludes the reference variables from the conditioning set. This type of exclusion is precisely what is needed to use $Z\perp Y|UX$ instead of $Z\cancel{\perp }Y|UXW$. We first state a general formulation of an instrumental entropic inequality that uses the unique information and we will then go back to the example of Figure 1B.

Proposition 2  

(Instrumental entropic inequality with maximum entropy unique information). Consider the variables $\mathbf{Z}$, X, Y, ${\mathbf{B}}_0$, and U, all observable except U a hidden variable. Consider that the causal structure is such that, for all $Z_i\in \mathbf{Z}$, no pair from $\{Z_i,X,Y\}$ is separable given that U is hidden. Consider an exclusive partition ${\mathbf{B}}_0=\{{\mathbf{B}}_1,{\mathbf{B}}_2\}$. Consider that the causal structure imposes the nontestable independencies $\mathbf{Z}\perp U|{\mathbf{B}}_0$ and $\mathbf{Z}\perp Y|UX{\mathbf{B}}_1$. These independencies result in the testable inequality

$$H\left(X\right|{\mathbf{B}}_0)\ge I(\mathbf{Z};X|{\mathbf{B}}_0)+I(\mathbf{Z};Y\backslash \backslash {\mathbf{B}}_2|{\mathbf{B}}_1,X).$$

(9)

Proof.  

The proof is analogous to the one of Proposition 1. Again, the departing quantity is $I(\mathbf{Z};U,X|{\mathbf{B}}_0)$. Using the chain rule to decompose $I(\mathbf{Z};U,X|{\mathbf{B}}_0)$ as the sum of $I(\mathbf{Z};U|{\mathbf{B}}_0)$ and $I(\mathbf{Z};X|{\mathbf{B}}_0,U)$, the independence $\mathbf{Z}\perp U|{\mathbf{B}}_0$ allows deriving $H\left(X\right|{\mathbf{B}}_0)$ as upper bound, as in Equation (2). For the lower bound, instead of Equation (3) that applies the DP inequality of conditional information, the DP inequality of unique information is applied:

$$\begin{array}{cc}\hfill I(\mathbf{Z};U,X|{\mathbf{B}}_0)\stackrel{\left(a\right)}{=}& I(\mathbf{Z};X|{\mathbf{B}}_0)+I(\mathbf{Z};U|{\mathbf{B}}_0,X)\stackrel{\left(b\right)}{\ge }I(\mathbf{Z};X|{\mathbf{B}}_0)+I(\mathbf{Z};U\backslash \backslash {\mathbf{B}}_2|{\mathbf{B}}_1,X)\stackrel{\left(c\right)}{\ge }\hfill \\ & I(\mathbf{Z};X|{\mathbf{B}}_0)+I(\mathbf{Z};Y\backslash \backslash {\mathbf{B}}_2|{\mathbf{B}}_1,X).\hfill \end{array}$$

(10)

Equality $\left(a\right)$ applies the chain rule of mutual information. Inequality $\left(b\right)$ applies the definition of the unique information as a contribution smaller than or equal to the conditional mutual information (Equation (6)). Inequality $\left(c\right)$ holds because $\mathbf{Z}\perp Y|UX{\mathbf{B}}_1$ by Lemma 3 implies the DP inequality of the unique information $I(\mathbf{Z};U\backslash \backslash {\mathbf{B}}_2|{\mathbf{B}}_1,X)\ge I(\mathbf{Z};Y\backslash \backslash {\mathbf{B}}_2|{\mathbf{B}}_1,X)$. Combining the upper bound $H\left(X\right|{\mathbf{B}}_0)$ and the lower bound $I(\mathbf{Z};X|{\mathbf{B}}_0)+I(\mathbf{Z};Y\backslash \backslash {\mathbf{B}}_2|{\mathbf{B}}_1,X)$ proves the testable inequality. □

In the example of Figure 1B, Proposition 2 applies with $\mathbf{Z}=Z$, ${\mathbf{B}}_1=\varnothing$, and ${\mathbf{B}}_2=W$ and results in

$$H\left(X\right|W)\ge I(Z;X\left|W\right)+I(Z;Y\backslash \backslash W|X).$$

(11)

The inequality of Equation (11) is causally imposed when the causal structure creates the independencies $Z\perp U|W$ and $Z\perp Y|UX$. These independencies would also exist if in Figure 1B variables X and W were connected by an arc $X\leftrightarrow W$. On the other hand, $W\to X$ would produce $Z\cancel{\perp }Y|UX$, and $X\to W$ would produce $Z\cancel{\perp }U|W$.

Comparing Equations (8) and (11), the first is derived with ${\mathbf{B}}_0=\varnothing$ and the second with ${\mathbf{B}}_0=W$, ${\mathbf{B}}_1=\varnothing$, and ${\mathbf{B}}_2=W$. To better appreciate the factors that determine their power, we can rewrite them passing the first term at the r.h.s. to the l.h.s.:

$$\begin{array}{cc}\hfill & H\left(X\right|Z)\ge I(Z;Y\left|X\right)\hfill \end{array}$$

(12a)

$$\begin{array}{cc}\hfill & H\left(X\right|Z,W)\ge I(Z;Y\backslash \backslash W\left|X\right).\hfill \end{array}$$

(12b)

In general, these inequalities are complementary. For example, consider that it is to be tested the compatibility of a data set with the causal structure in Figure 1B with no edge $Z-U$ and no edge $Z-Y$. This causal structure creates independencies $Z\perp U$, $Z\perp U|W$, and $Z\perp Y|UX$, and hence causally imposes both inequalities of Equations (8) and (11). Therefore, the violation of any of the two inequalities suffices to discard the causal structure. Comparing their form, Equation (12b) has a smaller or equal upper bound than Equation (12a), given the monotonicity of entropy under conditioning. However, it also has a smaller or equal lower bound, since the unique information is upper-bounded by the unconditional mutual information (Equation (6)). This means that, for a concrete data set that has been generated from another causal structure that does not impose the fulfillment of the inequalities, any of the two inequalities can be violated while the other is not, so that their use is complementary for causal inference. Using $Z\perp U|W$ instead of $Z\perp U$ allows decreasing the upper bound, but when W is a collider between Z and Y such that $Z\cancel{\perp }Y|UXW$, a smaller observable lower bound is derived with the unique information and $Z\perp Y|UX$.

3.2. Instrumental Entropic Inequalities with Multivariate Instrumental Sets

So far, the example of Figure 1B presented an application of Proposition 2 with a univariate instrument $\mathbf{Z}=Z$. However, to establish that the new type of inequalities of Proposition 2 contributes additional causal inference power to the standard instrumental entropic inequalities, we also need to examine the standard instrumental inequalities with multivariate instrumental sets that exist for the same causal structure. To do so, we first highlight three key elements of the structure of instrumental entropic inequalities.

The derivation of both Propositions 1 and 2 departs from the quantity $I(\mathbf{Z};U,X|{\mathbf{B}}_0)$. The first key element is that the two required types of independencies play separate roles in the derivation of instrumental inequalities: $\mathbf{Z}\perp U|{\mathbf{B}}_0$ is used to derive the observable upper bound, while the other independence is used to derive the observable lower bound. In more detail, $\mathbf{Z}\perp U|{\mathbf{B}}_0$ is used after $I(\mathbf{Z};U,X|{\mathbf{B}}_0)$ is separated into $I(\mathbf{Z};U|{\mathbf{B}}_0)$ and $I(\mathbf{Z};X|{\mathbf{B}}_0,U)$. The other required independence is used to derive the observable lower bound thanks to a DP inequality, which is applied to $I(\mathbf{Z};U|{\mathbf{B}}_0,X)$. The different DP inequality applied is what differentiates Propositions 1 and 2.

The second key element is better appreciated rewriting the inequalities of Propositions 1 and 2 passing the the first term of the r.h.s. to the l.h.s.:

$$\begin{array}{cc}\hfill & H\left(X\right|\mathbf{Z},{\mathbf{B}}_0)\ge I(\mathbf{Z};Y|{\mathbf{B}}_0,X)\hfill \end{array}$$

(13a)

$$\begin{array}{cc}\hfill & H\left(X\right|\mathbf{Z},{\mathbf{B}}_0)\ge I(\mathbf{Z};Y\backslash \backslash {\mathbf{B}}_2|{\mathbf{B}}_1,X).\hfill \end{array}$$

(13b)

Written like this, the two inequalities have upper bound $H\left(X\right|\mathbf{Z},{\mathbf{B}}_0)$, with a conditioning set $\{\mathbf{Z},{\mathbf{B}}_0\}$ that does not differentiate between $\mathbf{Z}$ and ${\mathbf{B}}_0$, which appear in different arguments of $\mathbf{Z}\perp U|{\mathbf{B}}_0$. Therefore, regarding the upper bound, there is an invariance under the exchange of variables between the causal discovery instrumental set $\mathbf{Z}$ and the conditioning set ${\mathbf{B}}_0$. Alternative instrumental entropic inequalities that would require independencies $\{\mathbf{Z},{\mathbf{B}}_4\}\perp U|{\mathbf{B}}_3$, with ${\mathbf{B}}_0=\{{\mathbf{B}}_3,{\mathbf{B}}_4\}$, would all lead to the same upper bound $H\left(X\right|\mathbf{Z},{\mathbf{B}}_0)$. Accordingly, instrumental inequalities with multivariate instrumental sets obtained under this invariance need to be considered in order to assess the additional causal inference power provided by the new type of inequalities introduced in Proposition 2.

We can see an example of multivariate instrumental set in Figure 1B, again focusing on the causal structure with no connections $Z-U$ and $Z-Y$. Variable W, which in the derivation of Equation (12b) is assigned to ${\mathbf{B}}_0=W$, can be assigned to the instrumental set, leading to the bivariate instrumental set $\mathbf{Z}=\{Z,W\}$ and to ${\mathbf{B}}_0=\varnothing$. The set $\{Z,W\}$ fulfills $\mathbf{Z}\perp U|{\mathbf{B}}_0$, namely $\{Z,W\}\perp U$. On the other hand, $\{Z,W\}$ does not fulfill the other independence condition $\mathbf{Z}\perp Y|UX{\mathbf{B}}_0$ required in Proposition 1, since $\{Z,W\}Y|UX$ due to the direct connection between W and Y.

This leads us to the third key element of the construction of instrumental entropic inequalities. In Propositions 1 and 2, there is a single set $\mathbf{Z}$ that appears in both independencies, namely ${\mathbf{Z}}^{\prime }=\mathbf{Z}$ in either $\mathbf{Z}\perp U|{\mathbf{B}}_0$ and ${\mathbf{Z}}^{\prime }\perp Y|UX{\mathbf{B}}_0$, or in $\mathbf{Z}\perp U|{\mathbf{B}}_0$ and ${\mathbf{Z}}^{\prime }\perp Y|UX{\mathbf{B}}_1$. However, this constraint is not necessary and can be relaxed. In the derivation of an observable lower bound, a DP inequality could be applied to any subset ${\mathbf{Z}}^{\prime }\subseteq \mathbf{Z}$. This is captured in the following Proposition. For later convenience, we now also consider multivariate variables $\mathbf{X}$, $\mathbf{Y}$, and $\mathbf{U}$:

Proposition 3  

(Chainlike instrumental entropic inequalities with multivariate instrumental sets). Consider variables $\mathbf{Z}$, $\mathbf{X}$, $\mathbf{Y}$, ${\mathbf{B}}_0$, and $\mathbf{U}$, all observable except $\mathbf{U}$ hidden variables. Consider that the causal structure is such that for at least a $Z_i\in \mathbf{Z}$ there is a nonempty subset ${\mathbf{X}}_i\in \mathbf{X}$ and ${\mathbf{Y}}_i\in \mathbf{Y}$ such that no pair in $\{Z_i,{\mathbf{X}}_i,{\mathbf{Y}}_i\}$ is separable with $\mathbf{U}$ hidden. Consider an exclusive partition in r parts of the multivariate instrumental set $\mathbf{Z}$ given by ${\mathbf{Z}}^{\left[r\right]}=\{{\mathbf{Z}}_0,{\mathbf{Z}}_1,\dots ,{\mathbf{Z}}_r\}$, with ${\mathbf{Z}}_0=\varnothing$. Consider that the causal structure imposes the nontestable independence $\mathbf{Z}\perp \mathbf{U}|{\mathbf{B}}_0$. This independence creates a nontestable instrumental entropic inequality

$$\begin{array}{c}\hfill H\left(\mathbf{X}\right|{\mathbf{B}}_0)\ge I(\mathbf{Z};\mathbf{X}|{\mathbf{B}}_0)+I(\mathbf{Z};\mathbf{U}|{\mathbf{B}}_0,\mathbf{X})=I(\mathbf{Z};\mathbf{X}|{\mathbf{B}}_0)+\sum _{j=1}^{r}I({\mathbf{Z}}_j;\mathbf{U}|{\mathbf{B}}_0,\mathbf{X},{\mathbf{Z}}^{[j-1]}),\end{array}$$

(14)

with ${\mathbf{Z}}^{[j-1]}=\{{\mathbf{Z}}_0,{\mathbf{Z}}_1,\dots ,{\mathbf{Z}}_{j-1}\}$, where nontestability is due to the nonestimable components of the lower bound. A nontrivial testable instrumental entropic inequality exists if for at least one term $I({\mathbf{Z}}_j;\mathbf{U}|{\mathbf{B}}_0,\mathbf{X},{\mathbf{Z}}^{[j-1]})$ at least one conditional independence exists that enables a data processing inequality to substitute that term by an estimable lower bound that contains some variables in $\mathbf{Y}$ and does not contain $\mathbf{U}$.

Proof.  

The derivation of the upper bound is the same as in Propositions 1 and 2. Starting from $I(\mathbf{Z};\mathbf{U},\mathbf{X}|{\mathbf{B}}_0)$ the chain rule is applied and the upper bound derived with $I(\mathbf{Z};\mathbf{U}|{\mathbf{B}}_0)=0$ thanks to $\mathbf{Z}\perp \mathbf{U}|{\mathbf{B}}_0$. The nonestimable lower bound follows from a direct application of the chain rule equality of conditional mutual information to separate $\{\mathbf{U},\mathbf{X}\}$ into the terms $I(\mathbf{Z};\mathbf{X}|{\mathbf{B}}_0)$ and $I(\mathbf{Z};\mathbf{U}|{\mathbf{B}}_0,\mathbf{X})$, followed by the chain rule to separate $\mathbf{Z}$ with the partition ${\mathbf{Z}}^{\left[r\right]}$. Finally, the proposition states that it suffices that at least one term of the sum in the lower bound can be replaced by at least one observable information term so that a nontrivial testable inequality is obtained, dropping all remaining terms in the lower bound that contain hidden variables. This replacement is possible when applying at least one DP inequality to at least a term $I({\mathbf{Z}}_j;\mathbf{U}|{\mathbf{B}}_0,\mathbf{X},{\mathbf{Z}}^{[j-1]})$. Of course, a testable instrumental inequality is also obtained if more than one term can be replaced by observable lower bounds. □

Note that Proposition 3 does not specify the form of the conditional independencies and associated DP inequalities applied to obtain observable lower bounds. New procedures to do so are to be specified in Section 3.4. The advantage of this formulation is that it distinguishes between the variables $\mathbf{Z}$ that appear in the condition $\mathbf{Z}\perp \mathbf{U}|{\mathbf{B}}_0$ and the subsets ${\mathbf{Z}}_j$ that are involved in the independencies associated with DP inequalities to derive observable terms in the lower bound. This explains why in Section 2.3 we defined causal discovery instrumental sets based only on $\mathbf{Z}\perp \mathbf{U}|{\mathbf{B}}_0$.

We can now reconsider the multivariate instrumental set of Figure 1B that assigns $\mathbf{Z}=\{Z,W\}$ and ${\mathbf{B}}_0=\varnothing$. As mentioned, $\{Z,W\}\perp U$ allows deriving an upper bound $H\left(X\right|Z,W)$. The reason why Proposition 1 could not be applied is that $\{Z,W\}Y|UX$ due to the direct connection between W and Y. However, Proposition 3 can be applied to the example of Figure 1B selecting a partition with $r=2$, ${\mathbf{Z}}^{\left[2\right]}=\{\varnothing ,\left\{Z\right\},\left\{W\right\}\}$. Then $I(Z,W;U|X)$ is decomposed into $I(Z;U|X)+I(W;U|X,Z)$. It now suffices to apply the DP inequality based on $Z\perp Y|UX$ to obtain a lower bound $I(Z;U|X)\ge I(Z;Y\left|X\right)$, and hence to obtain the instrumental entropic inequality

$$H\left(X\right)\ge I(Z,W;X)+I(Z;Y|X).$$

(15)

The test with this inequality subsumes both tests of Equations (8) and (11), derived from Propositions 1 and 2, respectively. This is because it can be rewritten as $H\left(X\right|Z,W)\ge I(Z;Y\left|X\right)$, and hence combines the upper bound of Equation (12b) and the lower bound of Equation (12a).

The consideration of instrumental inequalities with multivariate instrumental sets discards that the new type instrumental entropic inequality of Proposition 2 provides additional causal inference power in the case of Figure 1B. More generally, we show in Appendix J that for a multivariate instrumental set $\mathbf{Z}$ that fulfills an independence $\mathbf{Z}\perp \mathbf{U}|{\mathbf{B}}_0$, no additional power can be gained from tests that use only a subset of $\mathbf{Z}$ as instrumental set, removing the rest of variables or transferring some variables of $\mathbf{Z}$ into the conditioning set ${\mathbf{B}}_0$. However, a causal structure may be such that for $\mathbf{Z}=\{{\mathbf{Z}}_1,{\mathbf{Z}}_2\}$ it holds ${\mathbf{Z}}_2\perp \mathbf{U}|{\mathbf{B}}_0{\mathbf{Z}}_1$ and yet $\mathbf{Z}\cancel{\perp }\mathbf{U}|{\mathbf{B}}_0$. Or with the opposite perspective, using the notation of Proposition 2 with ${\mathbf{B}}_0=\{{\mathbf{B}}_1,{\mathbf{B}}_2\}$, the causal structure may be such that $\mathbf{Z}\perp \mathbf{U}|{\mathbf{B}}_0$, but $\{\mathbf{Z},{\mathbf{B}}_2\}\mathbf{U}|{\mathbf{B}}_1$. In this case, Proposition 2 can add causal inference power.

An example of this is illustrated in Figure 1C. With no direct connection between Z and U, the independence $Z\perp U|W_1{W}_2$ holds. On the other hand, $\{Z,W_1\}U|W_2$, $\{Z,W_2\}U|W_1$, and $\{Z,W_1,W_2\}U$. If any of these independencies existed, the upper bound $H\left(X\right|Z,W_1,W_2)$ would be equally obtained from the corresponding multivariate instrumental set, because of the invariance of the upper bound to exchanges between $\mathbf{Z}$ and ${\mathbf{B}}_0$. In those cases, the variables of $\{W_1,W_2\}$ included in the instrumental set could be marginalized instead of requiring the use of a unique information with both variables in the reference argument. However, since these other independencies do not exist, the instrumental inequality constructed from Proposition 2 using $Z\perp U|W_1{W}_2$ and $Z\perp Y|UX$ adds additional causal inference power with the test

$$H\left(X\right|W_1,W_2)\ge I(Z;X|W_1,W_2)+I(Z;Y\backslash \backslash \{W_1,W_2\}\left|X\right).$$

(16)

In this section, we have addressed the issue of whether the new type of entropic inequalities with unique information terms can add causal inference power when tested together with standard instrumental inequalities that use related multivariate instrumental sets. Our objective was to ensure that the new type of inequalities is not trivially subsumed. To do so, we have contemplated two factors that determine when a new inequality test adds causal inference power. First, that there is some hypothesized causal structure of interest that causally imposes the new type of inequality, possibly together with other inequalities. Second, that the form of the new inequality is such that a probability distribution can exist for which the new test is rejected while no other causally-imposed inequality test is simultaneously rejected. On the other hand, if the new inequality is such that when violated also another inequality is always violated, then it does not add additional power. In Appendix B, we provide a full formal statement of the if and only if conditions under which a new entropic inequality test adds additional power to a set of other tests.

By examining multivariate instrumental sets, we have provided an example in Figure 1C for which Proposition 2 provides additional causal inference power. More broadly, we have seen in Proposition 3 that a nontestable entropic inequality is associated with each instrumental set. The identification of a set of variables as an instrumental set implies the existence of an observable upper bound, and the lower bound can be decomposed as a sum of nonobservable information terms. Nontrivial testable entropic inequalities are obtained when finding observable lower bounds of some of these terms. Proposition 3 accommodates the inequalities derived in Propositions 1 and 2. Compared to those, it relaxes the condition of Propositions 1 and 2 that requires that no pair from $\{Z_i,X,Y\}$ is separable when U is hidden, for all $Z_i\in \mathbf{Z}$. This is because Proposition 3 does not cover only the specific application of a type of DP inequality associated with a specific conditional independence, contrarily to Propositions 1 and 2 that are linked to $\mathbf{Z}\perp Y|UX{\mathbf{B}}_0$ and $\mathbf{Z}\perp Y|UX{\mathbf{B}}_1$, respectively. Section 3.3, Section 3.4 and Section 3.5 will generalize procedures to convert nontestable instrumental entropic inequalities into testable ones.

3.3. Instrumental Entropic Inequalities with Mutual Information and Maximum Entropy Unique Information Terms: The Case with Two Data Processing Inequalities Applied

We have seen above that, when a single conditional independence is used to derive a lower bound, the unique information DP inequality only increases causal inference power when the variables in the reference argument of the unique information are not part of a valid instrumental set (i.e., no transfer from ${\mathbf{B}}_2$ as specified in Proposition 2 to $\mathbf{Z}$ creates a valid instrumental set). We now show that this limitation does not occur when two DP inequalities are used to add terms in the lower bound. We show that unique information terms can be added in the lower bound not only instead of conditional information terms, but in addition to them, resulting in an increase of causal inference power. We use this scenario to illustrate the procedure that will then be generalized to combine an arbitrary number of DP inequalities, comprising DP inequalities of conditional mutual information, unique information, and of minInf information terms, as will be introduced in Section 3.4.

Proposition 4  

(Instrumental entropic inequalities with conditional mutual information and unique information terms). Consider variables $\mathbf{Z}$, $\mathbf{X}$, $\mathbf{Y}=\{{\mathbf{Y}}_1,{\mathbf{Y}}_2\}$, ${\mathbf{B}}_0=\{{\mathbf{B}}_1,{\mathbf{B}}_2\}$, and $\mathbf{U}$, all observable except $\mathbf{U}$ hidden variables. Consider that the causal structure is such that for at least a $Z_i\in \mathbf{Z}$ there is a nonempty subset ${\mathbf{X}}_i\in \mathbf{X}$ and ${\mathbf{Y}}_i\in \mathbf{Y}$ such that no pair in $\{Z_i,{\mathbf{X}}_i,{\mathbf{Y}}_i\}$ is separable when $\mathbf{U}$ is hidden. Consider that the causal structure imposes the nontestable independencies $\mathbf{Z}\perp \mathbf{U}|{\mathbf{B}}_0$, $\mathbf{Z}\perp {\mathbf{Y}}_1|\mathbf{U}\mathbf{X}{\mathbf{B}}_0$, and $\mathbf{Z}\perp {\mathbf{Y}}_2|\mathbf{U}\mathbf{X}{\mathbf{B}}_1{\mathbf{Y}}_1\backslash \overline{\mathbf{Y}}$, with $\overline{\mathbf{Y}}\subseteq {\mathbf{Y}}_1$. These independencies result in the testable inequality

$$H\left(\mathbf{X}\right|{\mathbf{B}}_0)\ge I(\mathbf{Z};\mathbf{X}|{\mathbf{B}}_0)+I(\mathbf{Z};{\mathbf{Y}}_1|{\mathbf{B}}_0,\mathbf{X})+I(\mathbf{Z};{\mathbf{Y}}_2\backslash \backslash \{{\mathbf{B}}_2,\overline{\mathbf{Y}}\}|{\mathbf{B}}_1,\mathbf{X},{\mathbf{Y}}_1\backslash \overline{\mathbf{Y}}).$$

(17)

Proof.  

The upper bound is derived as in Proposition 1, given the independence $\mathbf{Z}\perp \mathbf{U}|{\mathbf{B}}_0$. To derive the lower bound, start with $I(\mathbf{Z};\mathbf{U}|{\mathbf{B}}_0,\mathbf{X})$ after extracting $I(\mathbf{Z};\mathbf{X}|{\mathbf{B}}_0)$ with the chain rule of mutual information:

$$\begin{array}{cc}\hfill & I(\mathbf{Z};\mathbf{U}|{\mathbf{B}}_0,\mathbf{X})\stackrel{\left(a\right)}{=}I(\mathbf{Z};\mathbf{U},{\mathbf{Y}}_1|{\mathbf{B}}_0,\mathbf{X})\stackrel{\left(b\right)}{=}I(\mathbf{Z};{\mathbf{Y}}_1|{\mathbf{B}}_0,\mathbf{X})+I(\mathbf{Z};\mathbf{U}|{\mathbf{B}}_0,\mathbf{X},{\mathbf{Y}}_1)\stackrel{\left(c\right)}{\ge }\hfill \\ \hfill & I(\mathbf{Z};{\mathbf{Y}}_1|{\mathbf{B}}_0,\mathbf{X})+I(\mathbf{Z};\mathbf{U}\backslash \backslash \{{\mathbf{B}}_2,\overline{\mathbf{Y}}\}|{\mathbf{B}}_1,\mathbf{X},{\mathbf{Y}}_1\backslash \overline{\mathbf{Y}})\stackrel{\left(d\right)}{\ge }I(\mathbf{Z};{\mathbf{Y}}_1|{\mathbf{B}}_0,\mathbf{X})+\hfill \\ & I(\mathbf{Z};{\mathbf{Y}}_2\backslash \backslash \{{\mathbf{B}}_2,\overline{\mathbf{Y}}\}|{\mathbf{B}}_1,\mathbf{X},{\mathbf{Y}}_1\backslash \overline{\mathbf{Y}}).\hfill \end{array}$$

(18)

Equality $\left(a\right)$ applies the DP inequality of conditional mutual information (Lemma 1) thanks to $\mathbf{Z}\perp {\mathbf{Y}}_1|\mathbf{U}\mathbf{X}{\mathbf{B}}_0$. Equality $\left(b\right)$ applies the chain rule equality of mutual information. Inequality $\left(c\right)$ holds from the definition of unique information, which has conditional mutual information as an upper bound (Equation (6)). Inequality $\left(d\right)$ applies the DP inequality of unique information with $\mathbf{Z}\perp {\mathbf{Y}}_2|\mathbf{U}\mathbf{X}{\mathbf{B}}_1{\mathbf{Y}}_1\backslash \overline{\mathbf{Y}}$. □

The examples of Figure 2A,B illustrate how the two types of DP inequalities are combined. For simplicity of the explanations, in contrast to Figure 1, in Figure 2 we only represent individual causal structures (no dashed connections). Our objective here is not to derive all existing inequalities in these graphs, but to illustrate the procedure to construct inequalities combining the two types of DP inequalities and to examine the additional causal inference power they can provide. In these causal structures, select the instrumental set $\mathbf{Z}=Z$ and conditioning set ${\mathbf{B}}_0=\{W_1,W_2\}$. With this selection, both causal structures impose inequalities of the type of Proposition 4.

In the example of Figure 2A, given ${\mathbf{B}}_0=\{W_1,W_2\}$, the upper bound $H\left(X\right|W_1,W_2)$ is derived with $Z\perp U|W_1{W}_2$. Following Proposition 4, the term with $Y_1$ in the r.h.s., $I(Z;Y_1|W_1,W_2,X)$, is obtained with $Z\perp Y_1|UX{W}_1{W}_2$, from the mutual information DP inequality. The term with $Y_2$, $I(Z;Y_2\backslash \backslash \{W_2,Y_1\}|W_1,X)$, is obtained with $Z\perp Y_2|UX{W}_1$, from the unique information DP inequality. The derivation corresponds to the assignment in Equation (17) of ${\mathbf{B}}_1=W_1$, ${\mathbf{B}}_2=W_2$, and $\overline{\mathbf{Y}}=Y_1$. Note that it is necessary to exclude $\{W_2,Y_1\}$ from the conditioning set because $Z\cancel{\perp }{Y}_2|UX{W}_1\mathbf{S}$, for any nonempty $\mathbf{S}\subseteq \{W_2,Y_1\}$. In Figure 2B, the upper bound and the term with $Y_1$ are derived in the same way. On the other hand, the term $I(Z;Y_2\backslash \backslash W_2|W_1,X,Y_1)$ results from $Z\perp Y_2|UX{W}_1{Y}_1$, with $\overline{\mathbf{Y}}=\varnothing$. The unique information is chosen with reference variable $W_2$ as opposed to the reference variable $\{W_2,Y_1\}$ for Figure 2A. This difference is due to $W_2\to Y_1$ in Figure 2A, which renders $Y_1$ a descendant of the collider $W_2$ in $Z-W_2-Y_2$, which needs to be removed from conditioning to create an independence. Note that while for simplicity Figure 2A,B show specific causal structures, Proposition 4 also applies under certain changes of these examples. The same inequalities apply in any graph in which arrows are assigned such that $W_1$ continues to be a noncollider in $Z-W_1\to U$ and $Z-W_1\to Y_1$, or if either $W_1-U$ or $W_1-Y_1$ is not present. The graphs could also comprise $W_1\to W_2$ or $Z-W_1\to Y_2$, with $W_1$ a noncollider.

We will now illustrate that instrumental inequalities of the type of Proposition 4 can add extra causal inference power. To do so, we show that, for example in Figure 2A, this type of instrumental entropic inequalities is not subsumed by any standard instrumental entropic inequality that uses a multivariate instrumental set. We here justify the additional causal inference power verifying that no standard instrumental inequality can jointly use the DP inequalities that introduce variables $Y_1$ and $Y_2$. This guarantees that it does not happen that the new type of instrumental inequality is only violated in cases in which already a standard instrumental inequality is violated. To complement this reasoning, in Appendix D we examine concrete numerical examples in which no rejection occurs for tests based on standard instrumental inequalities, while rejections are obtained when incorporating unique information terms as in Proposition 4.

Consider the multivariate instrumental sets that can be used as an alternative given the selection of the instrumental set $\mathbf{Z}=Z$ and conditioning set ${\mathbf{B}}_0=\{W_1,W_2\}$ in Figure 2A,B. We know from Section 3.2 that the same upper bound would hold for any valid instrumental set obtained from a transfer between these sets, namely with $\mathbf{Z}=\{Z,\mathbf{S}\}$, and ${\mathbf{B}}_0=\{W_1,W_2\}\backslash \mathbf{S}$, for $\mathbf{S}\subseteq \{W_1,W_2\}$. Given the direct connection between $W_1$ and U, no set containing $W_1$ fulfills the criterion $\mathbf{Z}\perp \mathbf{U}|{\mathbf{B}}_0$ to be an instrumental set. On the other hand, $\{Z,W_2\}\perp U|W_1$, so that $\mathbf{Z}=\{Z,W_2\}$ is a valid instrumental set with ${\mathbf{B}}_0=W_1$.

We now focus specifically on Figure 2A. Starting from $I(Z,W_2;U|W_1,X)$, we examine different partitions of $\mathbf{Z}=\{Z,W_2\}$ that can be selected to apply Proposition 3. One partition is ${\mathbf{Z}}^{\left[2\right]}=\{\varnothing ,\left\{Z\right\},\left\{W_2\right\}\}$, which results in terms $I(Z;U|W_1,X)$ and $I(W_2;U|W_1,X,Z)$. Given that $W_2$ has direct connections to $Y_1$ and $Y_2$, and hence it is nonseparable from them, no DP inequality can be applied to $I(W_2;U|W_1,X,Z)$. For $I(Z;U|W_1,X)$, the DP inequality associated with $Z\perp Y_2|UX{W}_1$ can be applied. However, it is not possible to introduce $Y_1$, which requires conditioning on $W_2$ in $Z\perp Y_1|UX{W}_1{W}_2$. The opposite partition ${\mathbf{Z}}^{\left[2\right]}=\{\varnothing ,\left\{W_2\right\},\left\{Z\right\}\}$ results in terms $I(W_2;U|W_1,X)$ and $I(Z;U|W_1,X,W_2)$. Again, no DP inequality is applicable to $I(W_2;U|W_1,X)$. We see that the remaining term $I(Z;U|W_1,X,W_2)$ corresponds to the term used as starting point when applying Proposition 4. This shows that in this example starting from the multivariate instrumental set comes back to $\mathbf{Z}=Z$, and ${\mathbf{B}}_0=\{W_1,W_2\}$.

The key element that leads the combination of the mutual information and unique information DP inequalities to add causal inference power is the intertwined requirements in the independencies $Z\perp Y_1|UX{W}_1{W}_2$ and $Z\perp Y_2|UX{W}_1$. In Figure 2A, conditioning on $W_2$ is necessary to separate Z from $Y_1$, since $W_2$ is a noncollider in $Z-W_2-Y_1$. At the same time, $W_2$ cannot appear in the conditioning set to separate Z and $Y_2$, since it is a collider in $Z-W_2-Y_2$. This means that $W_2$ cannot simply be marginalized to exploit jointly the two independencies $Z\perp Y_1|UX{W}_1{W}_2$ and $Z\perp Y_2|UX{W}_1$. It needs to first appear in the conditioning set (when applying the DP inequality of conditional mutual information) and then be excluded from the conditioning set (leading to the application of the unique information DP inequality).

This analysis of Figure 2A highlights the difference with the scenario addressed in Section 3.1, in which a single DP inequality was applied. With a single DP inequality, the unique information DP inequality can only contribute to increase causal inference power when the variables that appear in the reference argument of the unique information cannot be part of a valid instrumental set. On the other hand, when combining DP inequalities, it is the intertwined structure of the independencies associated with different types of DP inequalities what requires their combination. To further highlight this point, in Appendix C we compare in more detail Figure 2A,B. For Figure 2B, the inequality of the type of Proposition 4 derived with $\mathbf{Z}=Z$ and ${\mathbf{B}}_0=\{W_1,W_2\}$ does not add causal inference power to the instrumental inequality derived with $\mathbf{Z}=\{Z,W_2\}$ and ${\mathbf{B}}_0=W_1$ that relies only on the DP inequality of conditional mutual information. The key difference is that in Figure 2A conditioning on $W_2$ is necessary to create the independence between Z and $Y_1$, while in Figure 2B the independence $Z\perp Y_1|UX{W}_1$ also holds. This does not create the intertwined structure of $Z\perp Y_1|UX{W}_1{W}_2$ and $Z\perp Y_2|UX{W}_1$, which require respectively the conditioning on $W_2$ and non conditioning on $W_2$ (see Appendix C for details).

In this section, we have examined how the DP inequality of conditional mutual information and of unique information can be used sequentially to introduce new observable information terms in the lower bound of an instrumental entropic inequality. Note that the chainlike instrumental entropic inequality of Proposition 3 accommodates the use of Proposition 4. Proposition 3 indicates potential partitions of $\mathbf{Z}$ into ${\mathbf{Z}}^{\left[r\right]}=\{\varnothing ,{\mathbf{Z}}_1,\dots ,{\mathbf{Z}}_r\}$, while Proposition 4 describes a procedure to derive observable information terms that can be introduced in parallel starting separately from different summands $I({\mathbf{Z}}_j;\mathbf{U}|{\mathbf{B}}_0,\mathbf{X},{\mathbf{Z}}^{[j-1]})$ of the r.h.s. of Equation (14).

In the next section we will see that the sequential addition of observable information terms can be extended with a more general type of minInf DP inequalities. With Proposition 4, we have seen that the combination of the DP inequality of conditional mutual information and unique information allows sequentially including and then removing from the conditioning set variables that are required to create an independence between $\mathbf{Z}$ and ${\mathbf{Y}}_1$, but that preclude from creating an independence between $\mathbf{Z}$ and ${\mathbf{Y}}_2$. This is achieved because, while the DP inequality of conditional mutual information operates in the original joint distribution $P(\mathbf{Z},\mathbf{U},{\mathbf{B}}_0,\mathbf{X},\mathbf{Y})$, the DP inequality of unique information operates within the family of distributions that only preserve $P(\mathbf{Z},\mathbf{U},{\mathbf{B}}_1,\mathbf{X},{\mathbf{Y}}_1\backslash \overline{\mathbf{Y}},{\mathbf{Y}}_2)$ and $P(\mathbf{Z},{\mathbf{B}}_0,\mathbf{X},{\mathbf{Y}}_1)$. It is the exclusion of ${\mathbf{B}}_2={\mathbf{B}}_0\backslash {\mathbf{B}}_1$ and of $\overline{\mathbf{Y}}$ from $P(\mathbf{Z},\mathbf{U},{\mathbf{B}}_1,\mathbf{X},{\mathbf{Y}}_1\backslash \overline{\mathbf{Y}},{\mathbf{Y}}_2)$ what allows exploiting an independence with only ${\mathbf{B}}_1$ in the conditioning set instead of ${\mathbf{B}}_0$. With the same logic, further relaxations of which marginals are preserved will allow us to sequentially combine more DP inequalities.

As a last remark, so far the introduction of new types of instrumental entropic inequalities (Propositions 2 and 4) has been accompanied by the comparison to related standard intrumental entropic inequalities with multivariate instrumental sets. This was necessary to verify that the new entropic inequalities do provide additional causal inference power. The examples of Figure 1C and Figure 2A show that indeed additional causal inference power can be gained either because of the lack of validity of corresponding multivariate instrumental sets (Figure 1C), or because of the intertwinement between the conditioning sets that appear in different independencies, which requires the application of the unique information DP inequality (Figure 2A). In the next sections, we will not proceed in the same way, and instead we will exclusively focus in developing instrumental entropic inequalities that add more minInf information terms in the lower bound. The verification that this addition can further increase causal inference power follows from the same logic of these previous examples. Numerical examples will be provided in Appendix H to illustrate that the addition of more minInf terms together with unique information terms increases causal inference power. It is out of the scope of this work to provide a full taxonomy of when instrumental inequalities that exploit certain types of DP inequalities are subsumed by instrumental inequalities that only exploit a subset of those types of DP inequalities. Only in Appendix J, we derive a hierarchy between specific types of instrumental entropic inequalities with related instrumental sets.

3.4. Recursive Use of Data Processing Inequalities to Add Observable minInf Information Terms as Lower Bounds of Information Terms with Hidden Variables

We have identified the DP inequality of unique information as the key property that allows increasing causal inference power using unique information terms. This raises the question of whether analogous DP inequalities exist for other minInf information terms defined with other sets of constraints on the preserved marginals and if so, how to recursively use these DP inequalities to insert additional observable information terms into entropic inequalities. We now show that indeed there is such a DP inequality for a more general form of minInf information terms.

This section contains our core results of how to exploit minInf DP inequalities. We have used the instrumental entropic inequality to vertebrate our presentation, but in Section 3.6 we will describe a wider framework for the finding of new entropic inequalities and in Section 3.7 we will provide further examples of the applicability of the tools here developed. To help differentiate between general results and results specific of the instrumental inequality scenario, we continue to use a different notation of variables specific for the instrumental scenario, separate from the notation used for general results. We present a general DP inequality for minInf terms using the same notation of the DP inequalities of mutual information (Lemma 1) and unique information (Lemma 3). We then show how to iteratively combine minInf DP inequalities to add new observable terms into entropic inequalities. In Section 3.5, we will show how to apply these tools concretely to the instrumental scenario.

Proposition 5  

(Data processing inequality in predictor variables of minInf information terms preserving sets of marginals). Let $\overline{\mathbf{Z}}$, $\mathbf{D}$, ${\mathbf{D}}^{\prime }$, $\mathbf{E}$, and ${\mathbf{E}}_2$ be five nonoverlapping sets of variables. Consider a probability distribution $P(\overline{\mathbf{Z}},\mathbf{D},{\mathbf{D}}^{\prime },\mathbf{E},{\mathbf{E}}_2)$ and the family of distributions $\Delta P_{\mathbf{D}{\mathbf{D}}^{\prime }}$ that share the set of marginals $P(\overline{\mathbf{Z}},\mathbf{D},{\mathbf{D}}^{\prime },{\mathbf{O}}_1)$ and $P(\overline{\mathbf{Z}},{\mathbf{O}}_i)$ for $i=2,\dots ,m$, where ${\mathbf{O}}^{\left[m\right]}=\{{\mathbf{O}}_0,{\mathbf{O}}_1,\dots ,{\mathbf{O}}_m\}$ is a collection of subsets ${\mathbf{O}}_i\subseteq \{\mathbf{E},{\mathbf{E}}_2\}$ and ${\mathbf{O}}_0=\varnothing$. If the distribution $P(\overline{\mathbf{Z}},\mathbf{D},{\mathbf{D}}^{\prime },\mathbf{E},{\mathbf{E}}_2)$ is such that $\overline{\mathbf{Z}}{\perp }_P{\mathbf{D}}^{\prime }|\mathbf{D}{\mathbf{O}}_1$, then

$$\underset{Q\in \Delta P_{\mathbf{D}{\mathbf{D}}^{\prime }}}{min}{I}_Q(\overline{\mathbf{Z}};\mathbf{D},{\mathbf{D}}^{\prime },{\mathbf{E}}_2|\mathbf{E})=\underset{Q\in \Delta P_{\mathbf{D}}}{min}{I}_Q(\overline{\mathbf{Z}};\mathbf{D},{\mathbf{E}}_2|\mathbf{E})\ge \underset{Q\in \Delta P_{{\mathbf{D}}^{\prime }}}{min}{I}_Q(\overline{\mathbf{Z}};{\mathbf{D}}^{\prime },{\mathbf{E}}_2|\mathbf{E}),$$

(19)

where $\Delta P_{\mathbf{D}}$ is the family of distributions that preserve $P(\overline{\mathbf{Z}},\mathbf{D},{\mathbf{O}}_1)$ and $P(\overline{\mathbf{Z}},{\mathbf{O}}_i)$ for $i=2,\dots ,m$, and $\Delta P_{{\mathbf{D}}^{\prime }}$ is the family of distributions that preserve $P(\overline{\mathbf{Z}},{\mathbf{D}}^{\prime },{\mathbf{O}}_1)$ and $P(\overline{\mathbf{Z}},{\mathbf{O}}_i)$ for $i=2,\dots ,m$.

Proof.  

Given the chain rule of mutual information

$$\begin{array}{cc}\hfill \underset{Q\in \Delta P_{\mathbf{D}{\mathbf{D}}^{\prime }}}{min}{I}_Q(\overline{\mathbf{Z}};\mathbf{D},{\mathbf{D}}^{\prime },{\mathbf{E}}_2|\mathbf{E})=& \underset{Q\in \Delta P_{\mathbf{D}{\mathbf{D}}^{\prime }}}{min}\left[I_Q(\overline{\mathbf{Z}};{\mathbf{E}}_2|\mathbf{E})\right.+\hfill \\ \hfill & \left.I_Q(\overline{\mathbf{Z}};\mathbf{D}|\mathbf{E},{\mathbf{E}}_2)+I_Q(\overline{\mathbf{Z}};{\mathbf{D}}^{\prime }|\mathbf{E},{\mathbf{E}}_2,\mathbf{D})\right],\hfill \end{array}$$

(20)

and

$$\begin{array}{c}\hfill \underset{Q\in \Delta P_{\mathbf{D}}}{min}{I}_Q(\overline{\mathbf{Z}};\mathbf{D},{\mathbf{E}}_2|\mathbf{E})=\underset{Q\in \Delta P_{\mathbf{D}}}{min}\left[I_Q(\overline{\mathbf{Z}};{\mathbf{E}}_2|\mathbf{E})+I_Q(\overline{\mathbf{Z}};\mathbf{D}|\mathbf{E},{\mathbf{E}}_2)\right].\end{array}$$

(21)

Now consider a distribution that minimizes Equation (21), namely

$$\begin{array}{c}\hfill Q^{*}(\overline{\mathbf{Z}},\mathbf{D},\mathbf{E},{\mathbf{E}}_2)\equiv \arg \underset{Q\in \Delta P_{\mathbf{D}}}{min}{I}_Q(\overline{\mathbf{Z}};\mathbf{D},{\mathbf{E}}_2|\mathbf{E}).\end{array}$$

(22)

Construct $\overline{Q}(\overline{\mathbf{Z}},\mathbf{D},{\mathbf{D}}^{\prime },\mathbf{E},{\mathbf{E}}_2)\equiv P\left({\mathbf{D}}^{\prime }\right|\mathbf{D},{\mathbf{O}}_1)Q^{*}(\overline{\mathbf{Z}},\mathbf{D},\mathbf{E},{\mathbf{E}}_2)$. Given that $Q^{*}\in \Delta P_{\mathbf{D}}$, it preserves the marginals $P(\overline{\mathbf{Z}},{\mathbf{O}}_i)$ for $i=2,\dots ,m$ and $P(\overline{\mathbf{Z}},\mathbf{D},{\mathbf{O}}_1)$. Furthermore, $\overline{Q}$ by construction preserves $\overline{\mathbf{Z}}{\perp }_P{\mathbf{D}}^{\prime }|\mathbf{D}{\mathbf{O}}_1$, which means that it preserves $P(\overline{\mathbf{Z}},\mathbf{D},{\mathbf{D}}^{\prime },{\mathbf{O}}_1)$, and hence $\overline{Q}\in \Delta P_{\mathbf{D}{\mathbf{D}}^{\prime }}$, since all other constraints to preserve marginals are the same in $\Delta P_{\mathbf{D}{\mathbf{D}}^{\prime }}$ and $\Delta P_{\mathbf{D}}$. Since the first two terms in the sum of Equation (20) do not depend on ${\mathbf{D}}^{\prime }$ their minimization is the same in $\Delta P_{\mathbf{D}}$ or $\Delta P_{\mathbf{D}{\mathbf{D}}^{\prime }}$ and $Q^{*}(\overline{\mathbf{Z}},\mathbf{D},\mathbf{E},{\mathbf{E}}_2)$ minimizes their sum, which is equal to the one in Equation (21). By construction of $\overline{Q}$, the independence $\{\overline{\mathbf{Z}},\mathbf{E},{\mathbf{E}}_2\}\backslash {\mathbf{O}}_1{\perp }_{\overline{Q}}{\mathbf{D}}^{\prime }|\mathbf{D}{\mathbf{O}}_1$ holds and hence, using the weak union axiom of semi-graphoids for mutual information [25,43], also the independence $\overline{\mathbf{Z}}{\perp }_{\overline{Q}}{\mathbf{D}}^{\prime }|\mathbf{D}\mathbf{E}{\mathbf{E}}_2$ holds. This means that the last term in the sum of Equation (20) is zero for $\overline{Q}$. Therefore, $\overline{Q}$ minimizes the r.h.s. of Equation (20), which is equal to the r.h.s. of Equation (21), so that $\underset{Q\in \Delta P_{\mathbf{D}{\mathbf{D}}^{\prime }}}{min}{I}_Q(\overline{\mathbf{Z}};\mathbf{D},{\mathbf{D}}^{\prime },{\mathbf{E}}_2|\mathbf{E})$ is equal to $\underset{Q\in \Delta P_{\mathbf{D}}}{min}{I}_Q(\overline{\mathbf{Z}};\mathbf{D},{\mathbf{E}}_2|\mathbf{E})$, with the minima reached by $\overline{Q}(\overline{\mathbf{Z}},\mathbf{D},{\mathbf{D}}^{\prime },\mathbf{E},{\mathbf{E}}_2)$ and $Q^{*}(\overline{\mathbf{Z}},\mathbf{D},\mathbf{E},{\mathbf{E}}_2)$, respectively. Furthermore, monotonicity of mutual information guarantees that information can only decrease when removing variable $\mathbf{D}$ from $\{\mathbf{D},{\mathbf{D}}^{\prime }\}$, namely $I_{\overline{Q}}(\overline{\mathbf{Z}};{\mathbf{D}}^{\prime },{\mathbf{E}}_2|\mathbf{E})$ is smaller than or equal to $I_{\overline{Q}}(\overline{\mathbf{Z}};\mathbf{D},{\mathbf{D}}^{\prime },{\mathbf{E}}_2|\mathbf{E})$. Finally, $\underset{Q\in \Delta P_{{\mathbf{D}}^{\prime }}}{min}{I}_Q(\overline{\mathbf{Z}};{\mathbf{D}}^{\prime },{\mathbf{E}}_2|\mathbf{E})$ by definition is smaller than or equal to $I_{\overline{Q}}(\overline{\mathbf{Z}};{\mathbf{D}}^{\prime },{\mathbf{E}}_2|\mathbf{E})$. □

Proposition 5 encompasses Lemmas 1 and 3 as subcases. The DP inequality of conditional mutual information is subsumed with $m=1$, ${\mathbf{E}}_2=\varnothing$, and ${\mathbf{O}}_1=\mathbf{E}$, such that $P(\overline{\mathbf{Z}},\mathbf{D},{\mathbf{D}}^{\prime },{\mathbf{O}}_1)$ corresponds to the joint original distribution. The DP inequality of unique information is subsumed with $m=2$, ${\mathbf{E}}_2=\varnothing$, ${\mathbf{O}}_2=\mathbf{E}$, ${\mathbf{O}}_1\subset {\mathbf{O}}_2$. This results in a unique information $I(\overline{\mathbf{Z}};\mathbf{D},{\mathbf{D}}^{\prime }\backslash \backslash {\mathbf{D}}_2|{\mathbf{O}}_1)$, with ${\mathbf{D}}_2=\{{\mathbf{O}}_2\backslash {\mathbf{O}}_1\}$, as in Lemma 3.

Given this DP inequality for minInf information terms, we now describe how it can be used to iteratively add new observable information terms in a lower bound of a minInf information term containing as predictor hidden variables. We start with an example to gain some intuition of the procedure. For this purpose, we recap how the DP inequalities of mutual information and unique information are sequentially combined in Proposition 4, concretely in the example of Figure 2A. We then point out how a similar procedure can be used to sequentially combine more minInf DP inequalities, using Figure 2C as an example.

The first row of

Table 1. Sequential relaxation of preserved marginals when combining minInf DP inequalities. (i) Combination of the DP inequality of mutual information and of unique information in Figure 2A. (ii) Combination of the DP inequality of mutual information and of unique information in Figure 2C. (iii) Combination with an additional minInf DP inequality in Figure 2C.

	Prior Use of a DP Inequality	Relaxation of Preserved Marginals	Subsequent Use of a DP Inequality
(i)	$P(Z,U,Y_1,W_1,W_2,X)$	$\{P(Z,U,W_1,X),P(Z,Y_1,W_1,W_2,X)\}$	$\{P(Z,U,Y_2,W_1,X),P(Z,Y_1,W_1,W_2,X)\}$
	$Z\perp Y_1|UX{W}_1{W}_2$		$Z\perp Y_2|UX{W}_1$
(ii)	$P(Z,U,Y_1,W_2,W_3,X)$	$\{P(Z,U,W_3,X),P(Z,Y_1,W_2,W_3,X)\}$	$\{P(Z,U,Y_2,W_3,X),P(Z,Y_1,W_2,W_3,X)\}$
	$Z\perp Y_1|UX{W}_2{W}_3$		$Z\perp Y_2|UX{W}_3$
(iii)	$\{P(Z,U,Y_2,W_3,X),$	$\{P(Z,U,Y_2,X),P(Z,Y_2,W_3,X),$	$\{P(Z,U,Y_3,Y_2,X),P(Z,Y_2,W_3,X),$
	$P(Z,Y_1,W_2,W_3,X)\}$	$P(Z,Y_1,W_2,W_3,X)\}$	$P(Z,Y_1,W_2,W_3,X)\}$
	$Z\perp Y_2|UX{W}_3$		$Z\perp Y_3|UX{Y}_2$

summarizes how the relaxation of preserved marginals allows combining the DP inequalities of mutual information and unique information in Figure 2A to sequentially insert $Y_1$ and $Y_2$. The key aspect of this relaxation is that only the variables involved in the independence $Z\perp Y_2|UX{W}_1$ are preserved in the marginal $P(Z,U,W_1,X)$ that includes the hidden variable. This allows applying the unique information DP inequality to insert $Y_2$, while $Z\cancel{\perp }{Y}_2|UX{W}_1{W}_2$ does not allow applying the mutual information DP inequality. The second row of Table 1 summarizes the analogous procedure applied to Figure 2C to combine the DP inequalities of mutual information and unique information to sequentially insert $Y_1$ and $Y_2$. A more detailed examination of the corresponding instrumental entropic inequality that holds for Figure 2C will be examined in Section 3.5. Here, our interest is to motivate that the same procedure of relaxation of the preserved marginals allows applying a third minInf DP inequality to insert $Y_3$.

This is shown in the third row of Table 1. The preserved marginals $\{P(Z,U,Y_2,W_3,X),$$P(Z,Y_1,W_2,W_3,X)\}$ in the third column of (ii), which allow applying the DP inequality of unique information, are the departing set in the first column of (iii). Then a new relaxation of the marginals divides $P(Z,U,Y_2,W_3,X)$ into $\{P(Z,U,Y_2,X),P(Z,Y_2,W_3,X)\}$. This allows preserving in $P(Z,U,Y_2,X)$ only the variables involved in the independence $Z\perp Y_3|UX{Y}_2$, while $Z\cancel{\perp }{Y}_3|UX{W}_3{Y}_2$. The other marginal $P(Z,Y_1,W_2,W_3,X)$ is left unchanged during the relaxation. We then recognize in the structure of the marginals preserved after the iterative application of the relaxations the pattern of constraints of the families of distributions considered in Proposition 5, namely the fact that the hidden variables and conditioning variables involved in the subsequent conditional independence to be exploited are the only ones included together with $\overline{\mathbf{Z}}$ in the marginal distribution that plays the role of $P(\overline{\mathbf{Z}},\mathbf{D},{\mathbf{O}}_1)$. Accordingly, the minInf DP inequality of Proposition 5 is used to insert $Y_3$.

We now formalize how DP inequalities of minInf terms that are determined by sequential relaxations of the preserved marginals can be combined:

Theorem 1  

(Iterative addition of observable minInf information terms to lower bounds of unobservable minInf information terms). Consider nonoverlapping sets of variables $\overline{\mathbf{Z}}$, $\mathbf{E}$, and $\overline{\mathbf{U}}$ with all observable except $\overline{\mathbf{U}}$ hidden variables. For $k\ge 1$, consider a nonempty collection of observable nonoverlapping sets of variables ${\mathbf{A}}^{\left[k\right]}=\{{\mathbf{A}}_0,{\mathbf{A}}_1,\dots ,{\mathbf{A}}_k\}$, with ${\mathbf{A}}_0=\varnothing$. Consider a collection ${\overline{\mathbf{Z}}}^{\left[k\right]}=\{{\overline{\mathbf{Z}}}_0,{\overline{\mathbf{Z}}}_1,\dots ,{\overline{\mathbf{Z}}}_k\}$ and a collection ${\check{\mathbf{Z}}}^{\left[k\right]}=\{{\check{\mathbf{Z}}}_0,{\check{\mathbf{Z}}}_1,\dots ,{\check{\mathbf{Z}}}_k\}$ such that ${\overline{\mathbf{Z}}}_0=\overline{\mathbf{Z}}$, ${\check{\mathbf{Z}}}_0=\varnothing$, and ${\check{\mathbf{Z}}}_j\subset {\overline{\mathbf{Z}}}_{j-1}$, ${\overline{\mathbf{Z}}}_j\subseteq {\overline{\mathbf{Z}}}_{j-1}\backslash {\check{\mathbf{Z}}}_j$, for $j=1,\dots ,k$. Consider a collection ${\overline{\mathbf{U}}}^{\left[k\right]}=\{{\overline{\mathbf{U}}}_0,{\overline{\mathbf{U}}}_1,\dots ,{\overline{\mathbf{U}}}_k\}$ such that ${\overline{\mathbf{U}}}_0={\overline{\mathbf{U}}}_1=\overline{\mathbf{U}}$ and ${\overline{\mathbf{U}}}_j\subseteq {\overline{\mathbf{U}}}_{j-1}$, for $j=1,\dots ,k$. Consider the collections of sets of variables ${\overline{\mathbf{B}}}^{\left[k\right]}=\{{\overline{\mathbf{B}}}_0,{\overline{\mathbf{B}}}_1,\dots ,{\overline{\mathbf{B}}}_k\}$ and ${\mathbf{C}}^{\left[k\right]}=\{{\mathbf{C}}_0,{\mathbf{C}}_1,\dots ,{\mathbf{C}}_k\}$, with ${\overline{\mathbf{B}}}_0=\mathbf{E}$, ${\mathbf{C}}_0=\varnothing$, and iteratively constructed such that ${\overline{\mathbf{B}}}_1^{\prime }\subseteq \{{\mathbf{A}}_0,{\overline{\mathbf{B}}}_0\}=\mathbf{E}$, ${\overline{\mathbf{B}}}_1=\{{\overline{\mathbf{B}}}_1^{\prime },{\check{\mathbf{Z}}}_1\}$, ${\mathbf{C}}_1=\{{\mathbf{A}}_0,{\overline{\mathbf{B}}}_0,{\check{\mathbf{Z}}}_1\}=\{\mathbf{E},{\check{\mathbf{Z}}}_1\}$, and for $j>1$, ${\overline{\mathbf{B}}}_j^{\prime }\subseteq \{{\mathbf{A}}_{j-1},{\overline{\mathbf{B}}}_{j-1}\}$, ${\overline{\mathbf{B}}}_j=\{{\overline{\mathbf{B}}}_j^{\prime },{\check{\mathbf{Z}}}_j\}$, and ${\mathbf{C}}_j=\{{\mathbf{A}}_{j-1},{\overline{\mathbf{B}}}_{j-1},{\check{\mathbf{Z}}}_j\}$, so that ${\overline{\mathbf{B}}}_j\subseteq {\mathbf{C}}_j$, for $j=1,\dots ,k$. Consider a joint distribution $P(\overline{\mathbf{Z}},\overline{\mathbf{U}},{\mathbf{A}}^{\left[k\right]},\mathbf{E})$. Consider the family of distributions $\Delta P_{k-1}$ preserving $P({\overline{\mathbf{Z}}}_{k-1},{\overline{\mathbf{U}}}_{k-1},{\mathbf{A}}_{k-1},{\overline{\mathbf{B}}}_{k-1})$ and $P({\overline{\mathbf{Z}}}_j,{\mathbf{C}}_j)$, for $j=1,\dots ,k-1$. Consider the family of distributions $\Delta P_k$ preserving $P({\overline{\mathbf{Z}}}_k,{\overline{\mathbf{U}}}_k,{\mathbf{A}}_k,{\overline{\mathbf{B}}}_k)$ and $P({\overline{\mathbf{Z}}}_j,{\mathbf{C}}_j)$, for $j=1,\dots ,k$. If ${\overline{\mathbf{Z}}}_k{\perp }_P{\mathbf{A}}_k|{\overline{\mathbf{U}}}_k{\overline{\mathbf{B}}}_k$, then

$$\begin{array}{cc}\hfill \underset{Q\in \Delta P_{k-1}}{min}{I}_Q({\overline{\mathbf{Z}}}_{k-1};{\overline{\mathbf{U}}}_{k-1}|\mathbf{E},{\mathbf{A}}^{[k-1]},{\check{\mathbf{Z}}}^{[k-1]})\ge & \underset{Q\in \Delta P_k}{min}{I}_Q({\overline{\mathbf{Z}}}_k;{\mathbf{A}}_k|\mathbf{E},{\mathbf{A}}^{[k-1]},{\check{\mathbf{Z}}}^{\left[k\right]})+\hfill \\ & \underset{Q\in \Delta P_k}{min}{I}_Q({\overline{\mathbf{Z}}}_k;{\overline{\mathbf{U}}}_k|\mathbf{E},{\mathbf{A}}^{\left[k\right]},{\check{\mathbf{Z}}}^{\left[k\right]}).\hfill \end{array}$$

(23)

Proof.  

The proof is provided in Appendix E. □

Theorem 1 provides a way to iteratively add additional observable terms at the lower bound of nonobservable information terms with hidden variables. A full understanding of how it proceeds can be gained with its proof. In the rest of this section, we highlight its main properties, describing the transition from the exploitation of independence ${\overline{\mathbf{Z}}}_{k-1}{\perp }_P{\mathbf{A}}_{k-1}|{\overline{\mathbf{U}}}_{k-1}{\overline{\mathbf{B}}}_{k-1}$ in iteration $k-1$ to the exploitation of ${\overline{\mathbf{Z}}}_k{\perp }_P{\mathbf{A}}_k|{\overline{\mathbf{U}}}_k{\overline{\mathbf{B}}}_k$ in iteration k. We start with the simplest scenario, in which ${\overline{\mathbf{Z}}}_j=\overline{\mathbf{Z}}$, ${\overline{\mathbf{U}}}_j=\overline{\mathbf{U}}$, and ${\check{\mathbf{Z}}}_j=\varnothing$, for $j=1,\dots ,k$. This case already allows appreciating the core of the recursiveness. It corresponds to the scenario in which the DP inequalities being used all have the same target variable $\overline{\mathbf{Z}}$ and rely on the same set of hidden variables $\overline{\mathbf{U}}$. Given that ${\check{\mathbf{Z}}}_j=\varnothing$ for $j=1,\dots ,k$, the iterative construction of ${\overline{\mathbf{B}}}^{\left[k\right]}$ and ${\mathbf{C}}^{\left[k\right]}$ can be simplified to ${\overline{\mathbf{B}}}_1\subseteq \{{\mathbf{A}}_0,{\overline{\mathbf{B}}}_0\}=\mathbf{E}$, ${\mathbf{C}}_1=\{{\mathbf{A}}_0,{\overline{\mathbf{B}}}_0\}=\mathbf{E}$, and for $j>1$, ${\overline{\mathbf{B}}}_j\subset \{{\mathbf{A}}_{j-1},{\overline{\mathbf{B}}}_{j-1}\}$ and ${\mathbf{C}}_j=\{{\mathbf{A}}_{j-1},{\overline{\mathbf{B}}}_{j-1}\}$. The auxiliary variables ${\overline{\mathbf{B}}}_j^{\prime }$ are not needed when ${\check{\mathbf{Z}}}_j=\varnothing$ for $j=1,\dots ,k$ because their role is only to add ${\check{\mathbf{Z}}}_j$ in ${\overline{\mathbf{B}}}_j=\{{\overline{\mathbf{B}}}_j^{\prime },{\check{\mathbf{Z}}}_j\}$. Furthermore, for $j>1$, we have ${\overline{\mathbf{B}}}_j\subset \{{\mathbf{A}}_{j-1},{\overline{\mathbf{B}}}_{j-1}\}$ instead of ${\overline{\mathbf{B}}}_j\subseteq \{{\mathbf{A}}_{j-1},{\overline{\mathbf{B}}}_{j-1}\}$ because an equality ${\overline{\mathbf{B}}}_j=\{{\mathbf{A}}_{j-1},{\overline{\mathbf{B}}}_{j-1}\}$ leads to applying a DP inequality in step $j-1$ with the independence $\overline{\mathbf{Z}}{\perp }_P{\mathbf{A}}_{j-1}|\overline{\mathbf{U}}{\overline{\mathbf{B}}}_{j-1}$ and in step j with $\overline{\mathbf{Z}}{\perp }_P{\mathbf{A}}_j|\overline{\mathbf{U}}{\mathbf{A}}_{j-1}{\overline{\mathbf{B}}}_{j-1}$. These two steps can then be merged in a new step $j-1$ that jointly adds $\{{\mathbf{A}}_{j-1},{\mathbf{A}}_j\}$ given $\overline{\mathbf{Z}}{\perp }_P{\mathbf{A}}_{j-1}{\mathbf{A}}_j|\overline{\mathbf{U}}{\overline{\mathbf{B}}}_{j-1}$, based on the contraction axiom of semi-graphoids [25,43].

For this simplest scenario, we now highlight the core of the recursiveness. The independencies of step $k-1$ and k are $\overline{\mathbf{Z}}{\perp }_P{\mathbf{A}}_{k-1}|\overline{\mathbf{U}}{\overline{\mathbf{B}}}_{k-1}$ and $\overline{\mathbf{Z}}{\perp }_P{\mathbf{A}}_k|\overline{\mathbf{U}}{\overline{\mathbf{B}}}_k$, respectively. The family $\Delta P_{k-1}$ preserves $P(\overline{\mathbf{Z}},\overline{\mathbf{U}},{\mathbf{A}}_{k-1},{\overline{\mathbf{B}}}_{k-1})$ and $P(\overline{\mathbf{Z}},{\mathbf{C}}_j)$ for $j=1,\dots ,k-1$, and given that ${\mathbf{C}}_k=\{{\mathbf{A}}_{k-1},{\overline{\mathbf{B}}}_{k-1}\}$, it hence preserves $P(\overline{\mathbf{Z}},\overline{\mathbf{U}},{\mathbf{A}}_{k-1},{\overline{\mathbf{B}}}_{k-1})=P(\overline{\mathbf{Z}},\overline{\mathbf{U}},{\mathbf{C}}_k)$. In iteration $k-1$, variables ${\mathbf{A}}_{k-1}$ are introduced using Proposition 5 with $\overline{\mathbf{Z}}{\perp }_P{\mathbf{A}}_{k-1}|\overline{\mathbf{U}}{\overline{\mathbf{B}}}_{k-1}$. Here the variables $\{\mathbf{Z},\mathbf{D},{\mathbf{D}}^{\prime },{\mathbf{O}}_1,\mathbf{E},{\mathbf{E}}_2\}$ of Proposition 5 are assigned as $\{\overline{\mathbf{Z}},\overline{\mathbf{U}},{\mathbf{A}}_{k-1},{\overline{\mathbf{B}}}_{k-1},\{\mathbf{E},{\mathbf{A}}^{[k-2]}\},\varnothing \}$. The family $\Delta P_{k-1}$ plays the role of $\Delta P_{\mathbf{D}{\mathbf{D}}^{\prime }}$ in Equation (19). When moving from $\Delta P_{k-1}$ to $\Delta P_k$, the preservation of $P(\overline{\mathbf{Z}},\overline{\mathbf{U}},{\mathbf{A}}_{k-1},{\overline{\mathbf{B}}}_{k-1})=P(\overline{\mathbf{Z}},\overline{\mathbf{U}},{\mathbf{C}}_k)$ is loosen to the preservation of two of its marginals, namely $P(\overline{\mathbf{Z}},\overline{\mathbf{U}},{\overline{\mathbf{B}}}_k)$ and $P(\overline{\mathbf{Z}},{\mathbf{C}}_k)$. The first is a marginal because ${\overline{\mathbf{B}}}_k\subseteq {\mathbf{C}}_k$. The second is a marginal because $\overline{\mathbf{U}}$ is removed. Now $\overline{\mathbf{U}}$ only appears in $P(\overline{\mathbf{Z}},\overline{\mathbf{U}},{\overline{\mathbf{B}}}_k)$. The variables ${\mathbf{A}}_k$ are introduced analogously to ${\mathbf{A}}_{k-1}$, using Proposition 5 now with $\overline{\mathbf{Z}}{\perp }_P{\mathbf{A}}_k|\overline{\mathbf{U}}{\overline{\mathbf{B}}}_k$. Here $\{\mathbf{Z},\mathbf{D},{\mathbf{D}}^{\prime },{\mathbf{O}}_1,\mathbf{E},{\mathbf{E}}_2\}$ are assigned as $\{\overline{\mathbf{Z}},\overline{\mathbf{U}},{\mathbf{A}}_k,{\overline{\mathbf{B}}}_k,\{\mathbf{E},{\mathbf{A}}^{[k-1]}\},\varnothing \}$. The family $\Delta P_k$ plays the role of $\Delta P_{\mathbf{D}{\mathbf{D}}^{\prime }}$ in Equation (19). The second term at the r.h.s. of Equation (23) has the same form as the one at the l.h.s., replacing $k-1$ by k. Comparing the two independencies used in steps $k-1$ and k, ${\overline{\mathbf{B}}}_k$ used in $\overline{\mathbf{Z}}{\perp }_P{\mathbf{A}}_k|\overline{\mathbf{U}}{\overline{\mathbf{B}}}_k$ is a subset of ${\overline{\mathbf{B}}}_{k-1}$ used in $\overline{\mathbf{Z}}{\perp }_P{\mathbf{A}}_{k-1}|\overline{\mathbf{U}}{\overline{\mathbf{B}}}_{k-1}$, except for the possible addition of variables from ${\mathbf{A}}_{k-1}$. This follows the same pattern already seen in Figure 2A with $Z\perp Y_1|UX{W}_1{W}_2$ and $Z\perp Y_2|UX{W}_1$, where ${\mathbf{A}}_1=\left\{Y_1\right\}$, ${\mathbf{A}}_2=\left\{Y_2\right\}$, ${\overline{\mathbf{B}}}_1=\{X,W_1,W_2\}$, and ${\overline{\mathbf{B}}}_2=\{X,W_1\}$, or in Figure 2B with $Z\perp Y_1|UX{W}_1{W}_2$ and $Z\perp Y_2|UX{W}_1{Y}_1$, where ${\mathbf{A}}_1$, ${\mathbf{A}}_2$, and ${\overline{\mathbf{B}}}_1$ are the same, and ${\overline{\mathbf{B}}}_2=\{X,W_1,Y_1\}$.

We now provide an overview of the rest of scenarios. Concrete examples are described in Section 3.5 and in Appendix F. These other scenarios comprise cases in which ${\overline{\mathbf{Z}}}_j$ or ${\overline{\mathbf{U}}}_j$ are not constant for $j=1,\dots ,k$. The cases with non constant ${\overline{\mathbf{U}}}_j$, given that ${\overline{\mathbf{U}}}_j\subseteq {\overline{\mathbf{U}}}_{j-1}$, correspond to cases in which some of the hidden variables $\overline{\mathbf{U}}$ are marginalized before applying subsequent DP inequalities. This happens when in step j the variables ${\overline{\mathbf{U}}}_{j-1}\backslash {\overline{\mathbf{U}}}_j$ are colliders or descendants of colliders in paths that lead to ${\overline{\mathbf{Z}}}_{j}P{\mathbf{A}}_j|{\overline{\mathbf{U}}}_{j-1}{\overline{\mathbf{B}}}_j$ as opposed to ${\overline{\mathbf{Z}}}_j{\perp }_P{\mathbf{A}}_j|{\overline{\mathbf{U}}}_j{\overline{\mathbf{B}}}_j$. An example will be shown in Figure A2B.

Similarly, if ${\check{\mathbf{Z}}}_j=\varnothing$ for $j=1,\dots ,k$, the cases with non constant ${\overline{\mathbf{Z}}}_j$ for some j correspond to cases in which some target variables are marginalized to apply subsequent DP inequalities. This is because the relations ${\check{\mathbf{Z}}}_j\subset {\overline{\mathbf{Z}}}_{j-1}$, ${\overline{\mathbf{Z}}}_j\subseteq {\overline{\mathbf{Z}}}_{j-1}\backslash {\check{\mathbf{Z}}}_j$, for $j=1,\dots ,k$ simplify to ${\overline{\mathbf{Z}}}_j\subseteq {\overline{\mathbf{Z}}}_{j-1}$ when ${\check{\mathbf{Z}}}_j=\varnothing$ for $j=1,\dots ,k$. This happens in step j when ${\overline{\mathbf{Z}}}_{j-1}P{\mathbf{A}}_j|\overline{\mathbf{U}}{\overline{\mathbf{B}}}_j$, as opposed to ${\overline{\mathbf{Z}}}_j{\perp }_P{\mathbf{A}}_j|\overline{\mathbf{U}}{\overline{\mathbf{B}}}_j$, because the variables ${\overline{\mathbf{Z}}}_{j-1}\backslash {\overline{\mathbf{Z}}}_j$ have active paths reaching ${\mathbf{A}}_j$. An example will be shown in Figure A2C.

Finally, the case in which ${\check{\mathbf{Z}}}_j\ne \varnothing$ for some j covers cases in which conditioning on ${\check{\mathbf{Z}}}_j\subset {\overline{\mathbf{Z}}}_{j-1}$ is necessary to create the independence ${\overline{\mathbf{Z}}}_j{\perp }_P{\mathbf{A}}_j|\overline{\mathbf{U}}{\overline{\mathbf{B}}}_j$, with ${\check{\mathbf{Z}}}_j\subseteq {\overline{\mathbf{B}}}_j$. Accordingly, in step j the variables ${\check{\mathbf{Z}}}_j$ are moved from target variables to conditioning variables using a chain rule of the information terms. An example will be shown in Figure A2D. Furthermore, the marginalization of some variables in $\overline{\mathbf{U}}$, the marginalization of some variables in $\overline{\mathbf{Z}}$, and the conditioning on some ${\check{\mathbf{Z}}}_j$ can co-occur in the same step j, leading to the final general formulation of Theorem 1. The commonality to all scenarios is that in Equation (23), while the term at the l.h.s. is not observable, the first term at r.h.s. does not depend on any hidden variable and hence leads to an observable term by relaxing the preservation of $P({\overline{\mathbf{Z}}}_k,{\overline{\mathbf{U}}}_k,{\mathbf{A}}_k,{\overline{\mathbf{B}}}_k)$ to $P({\overline{\mathbf{Z}}}_k,{\mathbf{A}}_k,{\overline{\mathbf{B}}}_k)$. Furthermore, the second term of the r.h.s. has the same form as the term at the l.h.s., which means that Theorem 1 can be applied recursively.

3.5. Instrumental Entropic Inequalities with Sums of minInf Information Terms

We now show how to use Theorem 1 to create testable entropic inequalities from the nontestable inequalities of Proposition 3. Reexamining the terms $I({\mathbf{Z}}_j;\mathbf{U}|{\mathbf{B}}_0,\mathbf{X},{\mathbf{Z}}^{[j-1]})$, $j=1,\dots ,r$, that appear in the r.h.s. of Equation (14), we see that each of these terms can be the starting point to iterate the addition of observable information terms using Theorem 1. The application of Theorem 1 to a nonempty subset of a partition ${\mathbf{Z}}^{\left[r\right]}=\{\varnothing ,{\mathbf{Z}}_1,\dots ,{\mathbf{Z}}_r\}$ converts a nontestable instrumental entropic inequality from Proposition 3 into testable.

Proposition 6  

(Testable instrumental entropic inequalities from the iterative application of data processing inequalities to minInf information terms). Consider nonoverlapping sets of variables $\mathbf{Z}$, $\mathbf{X}$, ${\mathbf{B}}_{\mathbf{0}}$, and $\mathbf{U}$, all observable except $\mathbf{U}$ hidden variables. Consider that the joint distribution of these variables is generated from a causal structure that creates the independence $\mathbf{Z}\perp \mathbf{U}|{\mathbf{B}}_{\mathbf{0}}$, which leads to a nontestable entropic inequality of the form $H\left(\mathbf{X}\right|{\mathbf{B}}_{\mathbf{0}})\ge I(\mathbf{Z};\mathbf{X}|{\mathbf{B}}_{\mathbf{0}})+I(\mathbf{Z};\mathbf{U}|{\mathbf{B}}_{\mathbf{0}},\mathbf{X})$. Consider an exclusive partition in r parts of the instrumental set $\mathbf{Z}$ given by ${\mathbf{Z}}^{\left[r\right]}=\{\varnothing ,{\mathbf{Z}}_1,\dots ,{\mathbf{Z}}_r\}$, such that $I(\mathbf{Z};\mathbf{U}|{\mathbf{B}}_{\mathbf{0}},\mathbf{X})$ is separated in the sum of r nonestimable information terms $I({\mathbf{Z}}_k;\mathbf{U}|{\mathbf{B}}_0,\mathbf{X},{\mathbf{Z}}^{[k-1]})$, $k=1,\dots ,r$. Select ${\mathbf{E}}_k=\{{\mathbf{B}}_0,\mathbf{X},{\mathbf{Z}}^{[k-1]}\}$. Consider a nonoverlapping subset ${\tilde{\mathbf{Z}}}^{\left[q\right]}=\{\varnothing ,{\tilde{\mathbf{Z}}}_1,\dots ,{\tilde{\mathbf{Z}}}_q\}\subseteq {\mathbf{Z}}^{\left[r\right]}$, $0<q\le r$, such that each ${\tilde{\mathbf{Z}}}_l\in {\tilde{\mathbf{Z}}}^{\left[q\right]}$ corresponds to a different ${\mathbf{Z}}_i\in {\mathbf{Z}}^{\left[r\right]}$. Consider that for each ${\mathbf{Z}}_i\in {\tilde{\mathbf{Z}}}^{\left[q\right]}$ it is possible to iteratively apply Theorem 1 with an initial assignment of its inputs $\{\overline{\mathbf{Z}},\mathbf{E},\overline{\mathbf{U}}\}$ as $\{{\mathbf{Z}}_i,{\mathbf{E}}_i,{\mathbf{U}}_i\}$, with ${\mathbf{U}}_i\subseteq \mathbf{U}$. Accordingly, for each ${\mathbf{Z}}_i\in {\tilde{\mathbf{Z}}}^{\left[q\right]}$, it is possible to construct collections ${\mathbf{A}}_i^{\left[n_i\right]}$, ${\overline{\mathbf{Z}}}_i^{\left[n_i\right]}$, ${\check{\mathbf{Z}}}_i^{\left[n_i\right]}$, ${\overline{\mathbf{U}}}_i^{\left[n_i\right]}$, ${\overline{\mathbf{B}}}_i^{\left[n_i\right]}$, and ${\mathbf{C}}_i^{\left[n_i\right]}$, with $n_i>0$, which are associated with sets of independencies ${\overline{\mathbf{Z}}}_{ij}{\perp }_P{\mathbf{A}}_{ij}|{\overline{\mathbf{U}}}_{ij}{\overline{\mathbf{B}}}_{ij}$, for $j=1,\dots ,n_i$, which are imposed by the causal structure. This leads to the testable instrumental entropic inequality

$$\begin{array}{c}\hfill H\left(\mathbf{X}\right|{\mathbf{B}}_{\mathbf{0}})\ge I(\mathbf{Z};\mathbf{X}|{\mathbf{B}}_0)+\sum _{{\mathbf{Z}}_i\in {\tilde{\mathbf{Z}}}^{\left[q\right]}}\sum _{j=1}^{n_i}\underset{Q\in \Delta P_{ij}}{min}{I}_Q({\overline{\mathbf{Z}}}_{ij};{\mathbf{A}}_{ij}|{\mathbf{E}}_i,{\mathbf{A}}_i^{[j-1]},{\check{\mathbf{Z}}}_i^{\left[j\right]}),\end{array}$$

(24)

where each family of distributions $\Delta P_{ij}$ preserves $P({\overline{\mathbf{Z}}}_{ij},{\mathbf{A}}_{ij},{\overline{\mathbf{B}}}_{ij})$ and $P({\overline{\mathbf{Z}}}_{ik},{\mathbf{C}}_{ik})$, for $k=1,\dots ,j$.

Proof.  

Proposition 6 follows directly from the iterative application of Theorem 1 to a subset of the nonobservable information terms in the sum of Equation (14). In each case, the theorem is applied starting from a different set of variables $\{\overline{\mathbf{Z}},\mathbf{E},\overline{\mathbf{U}}\}$, namely $\{{\mathbf{Z}}_i,{\mathbf{E}}_i,{\mathbf{U}}_i\}$, where ${\mathbf{Z}}_i\in {\mathbf{Z}}^{\left[r\right]}$ corresponds to some ${\tilde{\mathbf{Z}}}_l\in {\tilde{\mathbf{Z}}}^{\left[q\right]}$. The variables in $\mathbf{U}$ not included in ${\mathbf{U}}_i$ are marginalized. Theorem 1 describes the properties that need to fulfill the collections ${\mathbf{A}}_i^{\left[n_i\right]}$, ${\overline{\mathbf{Z}}}_i^{\left[n_i\right]}$, ${\check{\mathbf{Z}}}_i^{\left[n_i\right]}$, ${\overline{\mathbf{U}}}_i^{\left[n_i\right]}$, ${\overline{\mathbf{B}}}_i^{\left[n_i\right]}$, and ${\mathbf{C}}_i^{\left[n_i\right]}$. The requirements $q>0$ and $n_i>0$ ensure that at least one observable information term is added in the lower bound, such that a nontrivial entropic inequality is testable. The form of the resulting testable inequality is determined by which independencies are imposed by the causal structure of interest, that is, which sets of independencies ${\overline{\mathbf{Z}}}_{ij}{\perp }_P{\mathbf{A}}_{ij}|{\overline{\mathbf{U}}}_{ij}{\overline{\mathbf{B}}}_{ij}$, for $i=1,\dots ,q$, $j=1,\dots ,n_i$ are combined to apply DP inequalities that add estimable information terms at the lower bound. □

The inequality of Proposition 6 encompasses the ones of Proposition 1, 2, and 4. It may be asked why the term $I(\mathbf{Z};\mathbf{X}|{\mathbf{B}}_0)$ is always separated before starting to apply DP inequalities. In fact, in the case of Proposition 1 where the standard DP inequality is applied, this is not a differentiating factor, since the r.h.s. of Equation (1) is equal to $I(\mathbf{Z};\mathbf{X},\mathbf{Y}|{\mathbf{B}}_0)$. However, when a DP inequality is applied in combination with relaxations of the constraints on the marginals to be preserved, this changes. As elaborated after the proof of Theorem 1 in Appendix E, in order to obtain a lower bound as tight as possible the constraints on the marginals should always be as strong as possible, while loose enough to allow the application of the subsequent DP inequalities. Given that the term $I(\mathbf{Z};\mathbf{X}|{\mathbf{B}}_0)$ is observable, a minimization after a relaxation of the constraints that would not preserve $P(\mathbf{Z},\mathbf{X},{\mathbf{B}}_0)$ results in an equal or smaller lower bound. This means that, to obtain the highest lower bound, DP inequalities should be applied starting with $I(\mathbf{Z};\mathbf{U}|{\mathbf{B}}_0,\mathbf{X})$, after the separation of $I(\mathbf{Z};\mathbf{X}|{\mathbf{B}}_0)$. Accordingly, we will further illustrate in Figure A2A that $\mathbf{X}$ and ${\mathbf{B}}_0$ play the same role in the derivation of an estimable lower bound.

We now examine in detail the application of Proposition 6 to the example of Figure 2C. The causal structure of Figure 2C is analogous to the one of Figure 2A, with some differences: It contains an additional predictor $Y_3$ and a new conditioning variable $W_3$. The conditioning variable $W_1$ has been removed for simplicity of the figure, but could be left as in Figure 2A with no qualitative effect in our reasoning. For simplicity of the explanation, we now focus on the construction of an instrumental entropic inequality with instrumental set $\mathbf{Z}=\left\{Z\right\}$ and conditioning set ${\mathbf{B}}_0=\{W_2,W_3\}$. We do so because this suffices to illustrate the iterative application of minInf DP inequalities with Proposition 6. See Appendix I for a more detailed analysis of this example. With $\mathbf{Z}=\left\{Z\right\}$ and ${\mathbf{B}}_0=\{W_2,W_3\}$, we apply Proposition 6 with ${\mathbf{A}}^{\left[n\right]}=\{\varnothing ,\left\{Y_1\right\},\left\{Y_2\right\},\left\{Y_3\right\}\}$, using the independencies $Z\perp Y_1|UX{W}_2{W}_3$, $Z\perp Y_2|UX{W}_3$, and $Z\perp Y_3|UX{Y}_2$. The derived entropic inequality is

$$\begin{array}{cc}\hfill H\left(X\right|W_2,W_3)\ge & I(Z;X,Y_1|W_2,W_3)+\underset{Q\in \Delta P_2}{min}{I}_Q(Z;Y_2|W_2,W_3,X,Y_1)+\hfill \\ & \underset{Q\in \Delta P_3}{min}{I}_Q(Z;Y_3|W_2,W_3,X,Y_1,Y_2),\hfill \end{array}$$

(25)

where $\Delta P_2$ preserves the marginals $\{P(Z,W_3,X,Y_2),P(Z,W_2,W_3,X,Y_1)\}$ and $\Delta P_3$ preserves the marginals $\{P(Z,X,Y_2,Y_3),P(Z,W_3,X,Y_2),P(Z,W_2,W_3,X,Y_1)\}$. The second term in the r.h.s. is the minInf term that corresponds to $I(Z;Y_2\backslash \backslash \{W_2,Y_1\}|W_3,X)$. Variable $Y_1$ is inserted thanks to $Z\perp Y_1|UX{W}_2{W}_3$, using the standard DP inequality of Lemma 1. Variable $Y_2$ is inserted thanks to $Z\perp Y_2|UX{W}_3$, using the DP inequality of unique information of Lemma 3. Finally, variable $Y_3$ is inserted thanks to $Z\perp Y_3|UX{Y}_2$, using a minInf DP inequality of the form of Proposition 5.

Our objective with this example was to illustrate the iterative insertion of estimable information terms in the lower bound. As mentioned above, an extended analysis of instrumental entropic inequalities for the causal structure of Figure 2C is presented in Appendix I. This extended presentation will cover alternative entropic inequalities that are derived with the multivariate instrumental set $\mathbf{Z}=\{Z,W_2,W_3\}$, when using different partitions to create chainlike instrumental entropic inequalities of the form of Equation (14). Concretely, the inequality of Equation (25) is subsumed by the one of Equation (A17e). Appendix I provides further evidence of the increase in causal inference power thanks to the addition of minInf terms, since they allow obtaining tighter lower bounds thanks to the combination of conditional independencies that cannot be jointly used in a standard instrumental inequality.

This completes our extension of instrumental entropic inequalities, from the standard form reviewed in Proposition 1, to the minInf instrumental entropic inequalities of Proposition 6. We have used the specific scenario of instrumental inequalities to vertebrate the presentation of our core contributions, namely the theoretical derivation of DP inequalities for minInf information terms (Proposition 5) and the iterative procedure to combine them (Theorem 1). In the rest of our Results, we will more broadly show how to apply these tools to derive other entropic inequalities apart from instrumental inequalities.

While our main contribution focuses on the theoretical derivation of the properties of minInf terms that render them useful for causal structure learning, in Appendix G we also discuss their estimation. As we explain, this estimation constitutes a non-convex minimization problem [44] and a general implementation is out of the scope of this work. Nonetheless, in Lemma A2 we recast minInf terms separating a convex and non-convex component of the minimization problem. We use this approach to extend the numerical examples presented in Appendix D, which show the gain in causal inference power obtained with Proposition 4. In this way, in Appendix H we also provide numerical examples in which it is only thanks to the addition of a second minInf term, like in Equation (25), that a rejection is obtained when testing the entropic inequality.

3.6. The Region of minInf Shannon Entropy Cones

In previous sections we have derived entropic inequalities with increased causal inference power by introducing minInf DP inequalities. We here more broadly reformulate this derivation from a geometrical perspective. In a geometrical perspective of entropy [22], entropy values associated with a set of variables $\overline{\mathbf{V}}=\{{\overline{V}}_1,{\overline{V}}_2,\dots ,{\overline{V}}_n\}$ are represented as a point in a ${\mathbb{R}}^{2^n}$ space. In more detail, given a joint distribution $P\left(\overline{\mathbf{V}}\right)$, for the set of indices $\left[n\right]=\{1,2,\dots ,n\}$ associated with the variables, an entropy value is obtained for each subset of indexes $S\subseteq \left[n\right]$. This entropy value corresponds to the joint entropy $H\left({\overline{\mathbf{V}}}_S\right)$ of the marginal probability distribution $P\left({\overline{\mathbf{V}}}_S\right)$. Given that the power set of subsets of $\left[n\right]$ contains $2^n$ subsets, a vector constructed with all entropy values $H\left({\overline{\mathbf{V}}}_S\right)$ lies in a ${\mathbb{R}}^{2^n}$ space. The region in this space containing all points obtainable from probability distributions, the entropy cone, forms a convex cone (Theorem 15.5 in [22]), but has an unknown explicit characterization. However, an approximation of this region is given by the Shannon entropy cone, which includes all points that comply with the following linear inequality constraints:

$$\begin{array}{cc}\hfill & H(\varnothing )=0,\hfill \end{array}$$

(26a)

$$\begin{array}{cc}\hfill & H\left({\overline{\mathbf{V}}}_T\right)\ge H\left({\overline{\mathbf{V}}}_S\right)\mathrm{if}S\subseteq T,\hfill \end{array}$$

(26b)

$$\begin{array}{cc}\hfill & I({\overline{\mathbf{V}}}_S;{\overline{\mathbf{V}}}_T|{\overline{\mathbf{V}}}_{S\cap T})\ge 0,\hfill \end{array}$$

(26c)

where S and T are two subsets of $\left[n\right]$. These inequalities are known as the basic inequalities [22,28] and can be expressed as linear inequalities only involving entropy terms, hence introducing constraints among different components of the entropy vectors. The basic inequalities impose requirements for any well-defined probability distribution, namely the nonnegativity of entropy (Equation (26a,b)), the monotonicity of entropy (Equation (26b)), and the nonnegativity of conditional mutual information (Equation (26c)), associated with the submodularity of entropy.

These basic inequalities are constraints that apply to any entropic vector created from a well-defined probability distribution. Furthermore, if a joint probability distribution of interest is generated under additional constraints, such as the compatibility with a certain causal structure, then the set of independencies induced by the causal structure adds extra equality constraints to the basic inequalities, namely in the form of conditional mutual information terms with zero values. In the presence of hidden variables, the cancelation of conditional mutual information terms involving the hidden variables is not verifiable. However, given the set combining the basic inequalities and the causally-induced equalities, it has been shown [14,24] that causally informative entropic inequalities, such as the standard instrumental entropic inequality, are derived by marginalization of the hidden variables to obtain inequalities that only involve observable variables. This marginalization has been algorithmically implemented using Fourier-Motzkin elimination, a standard linear programming algorithm for the elimination of variables from systems of inequalities [14].

The derivation of entropic inequalities with minInf terms can be formulated as an analogous marginalization problem, but starting from the region of a minInf Shannon entropy cone, which generalizes the region of the Shannon entropy cone from individual distributions to families of distributions sharing sets of constraints. In more detail, consider a minInf term

$$\begin{array}{c}\hfill \underset{Q\in \Delta P}{min}{I}_Q({\overline{\mathbf{V}}}_1;{\overline{\mathbf{V}}}_2|{\overline{\mathbf{V}}}_3),\end{array}$$

(27)

where ${\overline{\mathbf{V}}}_i$, $i=1,\dots ,3$ are sets of variables, without specification of which variables are observable or hidden. The family of distributions $\Delta P$ is defined as preserving a set of marginals from a joint original distribution $P\left(\overline{\mathbf{V}}\right)$, with $\{{\overline{\mathbf{V}}}_1,{\overline{\mathbf{V}}}_2,{\overline{\mathbf{V}}}_3\}\subseteq \overline{\mathbf{V}}$. The minimum within $\Delta P$ determines a set of distributions, at least one, that reach the minimum. Select a distribution $Q^{*}({\overline{\mathbf{V}}}_1,{\overline{\mathbf{V}}}_2,{\overline{\mathbf{V}}}_3)$ among the ones reaching the minimum and consider the region that its entropic vector can occupy. This region is restricted by basic inequalities (Equation (26)) and also by the constraints intrinsic to the definition of the minInf term in Equation (27). First, there is an additional constraint $I_{Q^{*}}({\overline{\mathbf{V}}}_1;{\overline{\mathbf{V}}}_2|{\overline{\mathbf{V}}}_3)\le I_P({\overline{\mathbf{V}}}_1;{\overline{\mathbf{V}}}_2|{\overline{\mathbf{V}}}_3)$, associated with the minimization, since $P\in \Delta P$ and $Q^{*}$ is a minimum. Second, there is a constraint $H_P\left({\overline{\mathbf{V}}}_S\right)=H_{Q^{*}}\left({\overline{\mathbf{V}}}_S\right)$ for any ${\overline{\mathbf{V}}}_S$ that appears in one of the marginal distributions preserved in $\Delta P$.

In the presence of additional constraints induced by a causal structure, the same constraints are imposed to P and $Q^{*}$ for those independencies associated with variables whose joint marginals are preserved. For example, for a causally-induced independence ${\overline{\mathbf{V}}}_1\perp {\overline{\mathbf{V}}}_2|{\overline{\mathbf{V}}}_3$, the constraints $I_P({\overline{\mathbf{V}}}_1;{\overline{\mathbf{V}}}_2|{\overline{\mathbf{V}}}_3)=0$ and $I_{Q^{*}}({\overline{\mathbf{V}}}_1;{\overline{\mathbf{V}}}_2|{\overline{\mathbf{V}}}_3)=0$ are imposed if $P({\overline{\mathbf{V}}}_1,{\overline{\mathbf{V}}}_2,{\overline{\mathbf{V}}}_3)$ is preserved in $\Delta P$. Overall, the entropic vector associated with the original distribution P and the vector associated with $Q^{*}$ are coupled. Furthermore, the set of constraints that characterizes the accessible region for entropic vectors also includes the minInf DP inequalities. This is because the derivation of these DP inequalities results from the definitions in terms of the minimization operator (see proofs of Lemma 3 and Proposition 5), that is, the DP inequality holds specifically for distributions at the minimum. Without imposing the DP inequalities as constraints to bound the region accessible to entropic vectors, the entropic vector of $Q^{*}({\overline{\mathbf{V}}}_1,{\overline{\mathbf{V}}}_2,{\overline{\mathbf{V}}}_3)$ could correspond to any distribution within $\Delta P$ fulfilling $I_{Q^{*}}({\overline{\mathbf{V}}}_1;{\overline{\mathbf{V}}}_2|{\overline{\mathbf{V}}}_3)\le I_P({\overline{\mathbf{V}}}_1;{\overline{\mathbf{V}}}_2|{\overline{\mathbf{V}}}_3)$, without corresponding to the minimum.

This type of coupling among entropic vectors does not uniquely result from constraints involving the original distribution P. Consider two families $\Delta P$ and $\Delta \tilde{P}$, with $\Delta \tilde{P}\subseteq \Delta P$, and a term $I_Q({\overline{\mathbf{V}}}_1;{\overline{\mathbf{V}}}_2|{\overline{\mathbf{V}}}_3)$ to be minimized. Consider distributions ${\tilde{Q}}^{*}$ and $Q^{*}$ that reach the minimum within $\Delta \tilde{P}$ and $\Delta P$, respectively. Since $\Delta \tilde{P}\subseteq \Delta P$, this means that there is a constraint $I_{{\tilde{Q}}^{*}}({\overline{\mathbf{V}}}_1;{\overline{\mathbf{V}}}_2|{\overline{\mathbf{V}}}_3)\ge I_{Q^{*}}({\overline{\mathbf{V}}}_1;{\overline{\mathbf{V}}}_2|{\overline{\mathbf{V}}}_3)$. Furthermore, there are also constraints $H_{{\tilde{Q}}^{*}}\left({\overline{\mathbf{V}}}_S\right)=H_{Q^{*}}\left({\overline{\mathbf{V}}}_S\right)$ for any ${\overline{\mathbf{V}}}_S$ that appears in the shared preserved marginals of $\Delta P$ and $\Delta \tilde{P}$. The same happens with the additional constraints induced by a causal structure. If both $\Delta P$ and $\Delta \tilde{P}$ preserve the marginal distribution containing the variables associated with a causally-imposed conditional independence, this results in information terms with zero values, creating a further coupling between entropic vectors corresponding to distributions in the two families.

Overall, there is a coupling between entropic vectors, comprising the one of the original probability distribution and those of the probability distributions defined in terms of minimizations within families that preserve certain sets of marginals. The resulting set of constraints includes constraints of different types. First, the basic inequalities of Equation (26), which apply to all distributions. Second, inequalities related to the definition of the minInf terms within a family of distribution preserving a set of marginals. This includes equalities between entropies when two families of distributions share marginals that are preserved. It also includes inequalities between information terms when they result from the minimization within families that are one a subset of the other. Third, it includes causally-induced constraints. This includes equalities (information terms with zero value) that apply to the original distribution and to any minInf distribution that preserves the joint marginal of variables involved in a conditional independence. This implies the DP inequalities of minInf terms.

The region accessible to the vectors compatible with a certain causal structure can thus be characterized in two dual ways. Given the selection of M minInf distributions resulting from M minimizations, one possibility is to describe the region as a set of $M+1$ interdependent entropic vectors within the ${\mathbb{R}}^{2^n}$ space of entropies. Another possibility is to define a ${\mathbb{R}}^{\left(2^n\right)(M+1)}$ space in which the entropic vectors of the original joint distribution P and of the M additional joint distributions associated with the minInf terms are appended. The latter representation constructs the minInf Shannon entropy cone.

A formal characterization of the region of minInf Shannon entropy cones is beyond the aim of this paper. However, the considerations above suggest how an algorithmic entropic characterization of causal structures [14,24] can be extended to exploit the constraints that exist in minInf Shannon entropy cones. Since the constraints involving minInf terms also constitute a linear system of equalities and inequalities, the procedure used to derive testable inequalities by the marginalization of hidden variables [14] can be extended to include minInf terms. Note that the minimization operations involved in the identification of the minInf distributions are not to be solved as part of the linear system. They are reflected in the system by the inclusion of the constraints associated with the definition of the minInf terms. This guarantees that, after the marginalization, the reduced system will contain entropic inequalities that express relations between estimable minInf terms, which can then be tested. The minInf instrumental entropic inequalities introduced in previous sections are one example of inequalities that would be obtained with this algorithmic approach. The implementation of this extended procedure is left for future work.

3.7. Other Types of Entropic Inequalities with minInf Information Terms

The implementation of a procedure to algorithmically derive causally informative inequalities with minInf terms is out of the scope of this work. However, we here point to two other well-known types of causally informative entropic inequalities that can be extended thanks to the minInf DP inequalities introduced in Section 3.4. We do not aim to provide a full presentation of these entropic inequalities, but to reframe them in a form that allows appreciating their extensions.

The first type of inequalities that we extend is the Groups-Decomposition (GD) inequalities [25,26]. We keep the notation of [26] to facilitate the comparison. This type of inequalities relates the information that a collection of variables has about a set of target variables $\mathbf{Y}$ with a weighted sum of the information contained in different groups defined as subsets of that collection. Two subtypes of GD inequalities were introduced in [25]. The existence of an inequality of the first subtype (GD1) imposes certain conditions of independence between the groups and determines the weights based on the overlap between them. The second subtype (GD2) has no requirements of independence, but only applies to collections and groups that constitute ancestral sets, that is, sets including all ancestors of their members. Several extensions were introduced in [26], comprising a relaxation of the required conditions of independence for GD1 and more flexibility in the configuration of groups for GD2. These extensions also included a generalization to allow for conditioning sets and the use of the DP inequalities of Lemmas 1 and 3 to derive testable GD inequalities from collections containing hidden variables.

We here present GD inequalities in a form that highlights how to apply the tools developed in Section 3.4, hence using minInf DP inequalities to increase their causal inference power. For simplicity, we restrict ourselves to GD1 inequalities, because the presentation of the second subtype is more mathematically heavy. We extend GD1 inequalities following Proposition 3 of [26], while an extension of GD2 analogously follows from their Theorem 2. For the purpose of this extension, we explicitly differentiate observable variables $\mathbf{V}$ and hidden variables $\mathbf{U}$. The inequality considers a target set of variables $\mathbf{Y}$ and a conditioning set $\mathbf{Z}$. It also considers a collection of variables ${\mathbf{B}}_{\left[n\right]}=[{\mathbf{B}}_1,\dots ,{\mathbf{B}}_n]$ formed by n groups, possibly overlapping. Each group can contain observable and hidden variables, such that ${\mathbf{B}}_i=\{{\mathbf{V}}_i,{\mathbf{U}}_i\}$. The GD1 inequality states that

$$\begin{array}{cc}\hfill & H\left(\mathbf{Y}\right|\mathbf{Z})\ge \sum _{i=1}^n{\displaystyle \frac{1}{d_{{\mathbf{B}}_i}}}I(\mathbf{Y};{\mathbf{B}}_i|\mathbf{Z})=\sum _{i=1}^n{\displaystyle \frac{1}{d_{{\mathbf{B}}_i}}}\left[I(\mathbf{Y};{\mathbf{V}}_i|\mathbf{Z})+I(\mathbf{Y};{\mathbf{U}}_i|\mathbf{Z},{\mathbf{V}}_i)\right],\hfill \end{array}$$

(28)

where $d_{{\mathbf{B}}_i}-1$ is the number of groups that intersect with ${\mathbf{B}}_i$. The inequality holds if the groups fulfill the following conditions of independence: Given disjoint partitions ${\mathbf{B}}_i=\{{\mathbf{B}}_i^{\left(1\right)},{\mathbf{B}}_i^{\left(2\right)}\}$, ${\mathbf{B}}_i^{\left(1\right)}\perp {\mathbf{B}}_j^{\left(1\right)}\backslash {\mathbf{B}}_i^{\left(1\right)}|\mathbf{Z}$ and ${\mathbf{B}}_i^{\left(2\right)}\perp {\mathbf{B}}_j\backslash {\mathbf{B}}_i^{\left(2\right)}|{\mathbf{B}}_i^{\left(1\right)}\mathbf{Z}$, for all $i\ne j$. For each group, the term $I(\mathbf{Y};{\mathbf{V}}_i|\mathbf{Z})$ can be estimated, while the term $I(\mathbf{Y};{\mathbf{U}}_i|\mathbf{Z},{\mathbf{V}}_i)$ is analogous to the terms with hidden variables that appear in Proposition 3. Ref. [26] used the DP inequalities of mutual information and unique information to derive tighter estimable lower bounds. An example of a causal structure for which this inequality is causally fulfilled is shown in Figure 3A. In this case, the structure of a Common Ancestors (CM) graph [24] is obtained after conditioning on Z, with all dependencies between observable variables mediated by hidden variables. A GD inequality exists selecting ${\mathbf{B}}_i={\mathbf{B}}_i^{\left(1\right)}=\left\{U_i\right\}$, such that ${\mathbf{B}}_i^{\left(1\right)}\perp {\mathbf{B}}_j^{\left(1\right)}|Z$ holds for all $i\ne j$. The standard DP inequality is applied to each group, thanks to $Y\perp V_i|Z{U}_i$, which leads to a testable GD inequality. Consider now the role of the terms $I(\mathbf{Y};{\mathbf{U}}_i|\mathbf{Z},{\mathbf{V}}_i)$ if the structure of Figure 3A is embedded as part of a larger causal structure for which other minInf DP inequalities can be applied. In that case, analogously to Figure 1 and Figure 2, the terms $I(\mathbf{Y};{\mathbf{U}}_i|\mathbf{Z},{\mathbf{V}}_i)$ can be the starting point to iteratively add estimable information terms at the lower bound of the inequality, with a procedure analogous to the one enabled by Propositions 3 and 6.

The second type of inequality to be generalized is the Information Causality (IC) inequality [23,31]. This inequality differs from the ones considered so far in that it contemplates a marginal scenario defined not only in terms of the presence of hidden variables, but also in terms of restrictions regarding which observable variables are jointly observable. The motivation of this marginal scenario is that the IC inequality was conceived to comprise also quantum systems. However, we here focus on its derivation for classical systems, since the consideration of quantum systems would require examining how to possibly adapt our results to the case in which Shannon entropy is substituted by von Neumann entropy [31]. In particular, we follow the generalization introduced by [31]. For classical systems, the derivation of this IC inequality can be understood in relation to the causal structure of Figure 3B. We have kept the notation used in [31] to facilitate a comparison with their derivations. The only exception is that we use U for the hidden variable, consistently with our previous derivations, while in [31] it corresponds to variable B. All variables $\mathbf{X}=\{X_1,\dots ,X_n\}$, $\mathbf{Y}=\{Y_1,\dots ,Y_n\}$, and M are observable, with U hidden. However, the marginal scenario is defined as imposing further constraints of observability, such that the variables in $\mathbf{Y}$ have mutually exclusive observability. Only marginal distributions of the form $p(\mathbf{X},M,Y_i)$ are observable, for each $Y_i\in \mathbf{Y}$. Accordingly, for an inequality to be testable, it can only contain information terms estimable from these marginals. We now present the IC inequality in a form that highlights how it can be extended based on the tools developed in Section 3.4:

$$\begin{array}{cc}\hfill & H\left(M\right)-H\left(\mathbf{X}\right)+\sum _{i=1}^{n}H\left(X_i\right)\stackrel{\left(a\right)}{\ge }\left[I(X_1;M)+\sum _{i=2}^{n}I(X_i;X_1,M)\right]+\hfill \\ & \left[I(X_1;U|M)+\sum _{i=2}^{n}I(X_i;U|X_1,M)\right]\stackrel{\left(b\right)}{\ge }\left[I(X_1;M)+\sum _{i=2}^{n}I(X_i;X_1,M)\right]+\hfill \\ & \left[I(X_1;Y_1|M)+\sum _{i=2}^{n}I(X_i;Y_i|X_1,M)\right]+\left[I(X_1;U|M,Y_1)+\sum _{i=2}^{n}I(X_i;U|X_1,M,Y_i)\right].\hfill \end{array}$$

(29)

Note that the selection of $X_1$ is arbitrary and without loss of generality it can be replaced by any other variable in $\mathbf{X}$. The detailed derivation of inequality $\left(a\right)$ can be found in Equations (16)–(21) of [31] and follows from Lemma 2 in [24]. Apart from basic properties of entropy, the derivation relies on the causally-imposed independence $\mathbf{X}\perp U$. On the other hand, inequality $\left(b\right)$ is derived thanks to the DP inequalities of mutual information associated with $X_1\perp Y_1|UM$ and $X_i\perp Y_i|UM{X}_1$, which provide observable lower bounds of the terms $I(X_1;U|M)$ and $I(X_i;U|X_1,M)$. More generally, these terms can be the starting point to iteratively add estimable information terms at the lower bound of the inequality, with a procedure analogous to the one enabled by Propositions 3 and 6. To highlight this, we have kept in the lower bound the nonestimable information terms that contain U and that are dropped in the final testable inequality of [31]. If the structure of Figure 3B is embedded as part of a larger causal structure such that other types of minInf DP inequalities can be applied, an iterative addition of observable information terms in the lower bound can proceed as in Proposition 6.

4. Discussion

In this work we have explored how causally informative entropic inequalities can be extended to contain minimum information (minInf) entropic terms. We have first examined how to combine the standard data processing (DP) inequality and a DP inequality for the maximum entropy unique information [26,29] to create new and tighter instrumental entropic inequalities [24]. In this way, we identified a procedure to recursively combine different types of DP inequalities to introduce additional observable terms in the lower bound of information terms that contain hidden variables. We then introduced a DP inequality for a general type of minInf information terms defined by information minimization within families of distributions that preserve sets of marginals shared with the original distribution. We have shown how to recursively apply these DP inequalities to exploit sets of independencies with different conditioning sets. We then have used this procedure to build minInf instrumental entropic inequalities that provide additional causal inference power. While our development of causally informative entropic inequalities uses as vector for its derivation the instrumental causal scenario, the procedure presented in Theorem 1 to recursively combine DP inequalities is general. We have exemplified this for two other types of entropic inequalities, namely the Groups-Decomposition inequality [25,26], and the Information Causality inequality [23,31].

More generally, we have also reframed the use of minInf entropic terms to derive causally informative entropic inequalities from a geometrical perspective [22]. Entropic inequalities can be systematically derived as a marginalization problem [14]. To derive causally informative inequalities that only involve entropic terms of the original distribution, this marginalization operates on the set of constraints that defines the Shannon entropy cone [22], in combination with the additional conditional independence constraints imposed by the causal structure. We have indicated that, in order to incorporate minInf entropic terms in this geometrical approach, Shannon entropy cones can be extended to not only contain entropic terms from the original distribution, but to jointly combine the minInf entropic terms associated with a set of families of distributions that share sets of marginals with it. The derivation of causally informative inequalities with minInf terms can then proceed analogously as a marginalization problem, now departing from a set of constraints that comprises those that the conditional independencies in the causal structure impose to the minInf entropic terms, such as the minInf DP inequalities we have derived. While we have conceptualized this procedure, its implementation in a linear programming algorithm [14,22] remains out of the scope of this work.

For the instrumental causal scenario, we have provided a range of examples of causal structures for which causal inference power is increased with the new tests, through the examples of the Figures in the main text and further examples in Appendix F and Appendix I. With an increasing number of causally informative inequality tests, an important question is how to determine minimal sets of tests that preserve the overall causal inference power. For this purpose, we have stated a criterion to determine when a new inequality test can add causal inference power to a set of tests (Appendix B). We have used this criterion to derive a hierarchy for instrumental inequalities with multivariate instrumental sets that encompasses the comparison of certain types of tests. On the other hand, we have also provided examples for which various minInf entropic inequalities are complementary among them, potentially providing each additional power.

Our contribution has focused on the derivation of causally informative minInf entropic inequalities for which the involved families of distributions are determined by constraints on shared marginals. Alternatively, the minInf terms could be determined comprising other types of constraints. Specifically for the unique information term developed within the decomposition of mutual information into redundant, unique, and synergistic components, alternative approaches [39,41,45] define unique information within families that combine other constraints. Future work should determine if alternative entropic inequalities are derived with these measures and the degree to which they contribute to increase causal inference power.

Our approach can also benefit the study of causal interactions in dynamical systems [14,46]. High-dimensional multivariate dynamical processes appear in many domains of interest, such as brain dynamics, e.g., [47,48,49] or econometrics [50,51]. For time-series, methods of causal inference both in the temporal [52] and spectral domain [53,54] predominantly rely on the detection of conditional independencies between observable signals, and hence are affected by hidden variables due to the partial observability of complex systems. Our treatment of the instrumental causal scenario suggests that our methods can help in these cases, given that concatenated instrumental-like causal motifs frequently appear in temporal dynamics, as represented for example by autoregressive moving-average models [52]. The unique information measure has already been applied to neural data, e.g., [55,56]. However, applicability of entropic inequalities with minInf terms to dynamical systems will require further study of the scalability of our methods to high-dimensional data. The applicability of entropic inequalities to time-series will also need to incorporate further assumptions such as stationarity [57] to make the approach operational.

Beyond applications to classical systems, it remains to be seen how to adapt the formulation of minInf entropic inequalities for causal inference in the quantum domain [16,31,58,59]. In our work, we have considered how to extend Information Causality inequalities [23] using minInf DP inequalities, but we focused on their formulation for classical systems [31]. Quantum mutual information defined in terms of von Neumann entropy [60] fulfills the standard DP inequality as well as the chain rule property, which is involved in the derivation of Theorem 1. Nonetheless, future work is required to pursue an adaptation to quantum systems.

In this work we have focused on causally informative entropic inequalities. We have mostly focused on the instrumental causal scenario to introduce new minInf entropic inequalities and examine them jointly with standard inequalities constructed with multivariate instrumental sets. More broadly, an important question regards the embedding of this type of causally informative entropic inequalities with other approaches to test the compatibility of causal structures with data [61,62,63]. The fact that the new inequalities appear as constraints only in the extended minInf Shannon entropy cones indicates that they cannot be reduced to constraints in the standard Shannon entropy cone of the original joint distribution. However, a separate question is how to construct minimal sets of inequality constraints with equivalent inference power, and how these sets would combine or prioritize the use of minInf tests and tests that operate in an inflated standard Shannon entropy cone [64]. The integration of different families of tests under the criterion that determines when the addition of a new test increases the causal inference power of a set of tests stands as a goal to determine the boundaries of distinguishably between causal structures from data.