Towards Moment-Constrained Causal Modeling ^†

Abstract

The fundamental problem with causal inference involves discovering causal relations between variables used to describe observational data. We address this problem within the formalism of information field theory (IFT). Specifically, we focus on the problems of bivariate causal discovery (X → Y, Y → X) from an observational dataset $(X,Y)$. The bivariate case is especially interesting because the methods of statistical independence testing are not applicable here. For this class of problems, we propose the moment-constrained causal model (MCM). The MCM goes beyond the additive noise model by exploiting Bayesian hierarchical modeling to provide non-parametric reconstructions of the observational distributions. In order to identify the correct causal direction, we compare the performance of our newly-developed Bayesian inference algorithm for different causal directions ($X\to Y$, Y → X) by calculating the evidence lower bound (ELBO). To this end, we developed a new method for the ELBO estimation that takes advantage of the adopted variational inference scheme for parameter inference.

1. Introduction

Since Pearl’s pioneering works on causality [1,2], various new methods have been developed for the study of causal inference (for example, see [3] for a complete overview). In particular, the case of bivariate causal inference remains of central interest because the traditional interventional methods are not applicable in this regime. The problem of inferring causal relations from observational data is usually tackled following one of three main approaches [4]. In the first approach, often referred to as the constraint-based approach, the correct causal graph is identified by analyzing conditional independence in the observational data distribution [5,6,7]. Methods that follow this approach can recover the true causal directed acyclic graph (DAG) up to its Markov equivalence class under some technical assumptions, but are inapplicable in the bivariate case, where no conditional-independence relationship is available.

The second approach, referred to as the score-based approach, fits the observational data by searching in the space of all possible DAGs of a given size and assigning a score to the result. This score is used to infer the correct causal relations [8], but again can only identify the causal graph up to its Markov equivalence class. Therefore, both constrained-based and score-based methods are not suited to perform causal inference in the bivariate case. However, combined constraint and score-based approaches that use generative neural networks (see Goudet et al. [9]) are also able to infer causal directions in the bivariate case.

The third and final approach is the asymmetry-based approach. Asymmetry-based methods exploit the asymmetries between the cause and the effect variables to reconstruct the underlying causal relations. These methods either evaluate the algorithm complexity of the two possible conditional distributions $p\left(y\right|x,$ X → Y$,\mathcal{M})$ and $p\left(x\right|y,$ Y → X$,\mathcal{M})$ resulting from splitting the joint probability density $p(x,y)$ given a model $\mathcal{M}$ [10,11,12,13] or check for the independence of the conditional probability of the effect given the cause from the cause’s independent probability distribution (information geometric causal inference [14,15]). This work most resembles the asymmetry-based approach described above. Here, we continue the series of works [16,17] that exploit the Information Field Theory (IFT) formalism [18,19,20,21] for causal inference, with two important novelties. First, we use the geometric variational inference (geoVI) approach as the inference tool for our hierarchical model. This has the advantage that the final covariance estimate of the posterior is better tailored to the problem and therefore for our evidence estimation, described in Section 3.7. Second, we propose a new causal model that can extend beyond the additive noise models (ANMs) formulation and allow for more flexible noise parameterization, while explicitly breaking the symmetry between cause and effect (Section 3.3).

2. Causal Inference and Bayesian Generative Models

In this section, we state the bivariate problem (Section 2.1), make a connection to the IFT formalism (Section 2.2), and then introduce structural causal models (SCMs) (Section 2.3).

2.1. The Bivariate Problem

The bivariate problem is of central importance in causal inference. Given a DAG representing a causal relation between N random variables $X_1,\dots ,X_n$, the joint distribution factorizes as

$$P(X_1,\dots ,X_n)=\prod _{i=1}^{N}P\left(X_i\right|{\mathrm{parent}}_{X_i}),$$

(1)

which is the Markov condition described by Pearl [1]. In fact, this shows that if we can characterize this factorization in such a way that the conditional probabilities are “smoother” or “simpler” for the true causal direction, we can recover the complete causal graph [13,22]. For this reason, i.e., to determine the correct causal structure of a complicated DAG, we only need to determine the parent cause direction, we can restrict ourselves to the bivariate X → Y and Y → X problems. The bivariate problem can be stated as follows. Given a dataset formed by pairs $d=(x,y)$ composed by realizations $x=(x_1,\dots ,x_N)\in {\mathcal{D}}_x$ and $y=(y_1,\dots ,y_N)\in {\mathcal{D}}_y$ of the random variables X and Y, respectively, predict the underlying causal direction choosing between X → Y or Y → X.

2.2. Causal Inference and IFT

We note that the structure of Equation (1) resembles Bayesian hierarchical generative modeling. This similarity is made even stronger if we consider the so-called structural causal models (SCMs) (see Section 6.2 of [3,23]). The SCM is composed of a DAG, which uniquely identifies the causal directions between a set of given variables, and a function f, which we refer to as the causal mechanism, and which links the causes to their effects within the DAG. In addition to the cause–effect pairs, the SCM is also supplemented by a set of noise variables $\xi$. Then the bivariate problem can be stated as

$$Y=f(X,\xi ).$$

(2)

Under certain conditions, this relation can uniquely identify the correct causal mechanism [1]. We stress how Equation (2) can easily be represented by a hierarchical generative model, once the split from Equation (1) is given. Since generative models can be directly related to inference (see, e.g., Enßlin [21]), we will use the framework of information field theory (IFT). IFT is Bayesian inference for the fields. Here, the fields we need to infer from noisy data are the probability distributions of X, Y, and their conditionals, given some model $\mathcal{M}$. For details on our inference scheme, we refer to Section 3.6. The problem of detecting the true underlying causal direction reduces to probing all possible splits in Equation (1) and selecting the one with the highest score, given by our evidence estimate (see Section 3.7).

2.3. SCMs and Bayesian Generative Modeling

Combining Bayesian generative modeling and IFT, it is possible to reproduce complicated densities non-parametrically, given some prior assumptions on their smoothness [24,25]. As discussed in Section 1, some asymmetry-based models, including the novel causal model we propose in this work (Section 3.3), exploit the idea that the conditional distributions are somewhat “simpler” in the correct causal direction. It is then natural to generate these probability distributions directly and then try to find their correspondent SCMs. In particular, if we want to produce the conditional distribution for the X → Y causal direction, we need to define a model $\mathcal{M}$ to generate $p\left(y\right|x,$X → Y$,\mathcal{M})$. We can then relate this distribution to an SCM of the form described in Equation (2) by considering

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill p\left(y\right|x,X\to Y,\mathcal{M})& ={\int }_{{\mathcal{D}}_{\xi }}p\left(y\right|x,\xi ,X\to Y,\mathcal{M})\mathcal{P}\left(\xi \right)\mathrm{d}\xi ={\int }_{{\mathcal{D}}_{\xi }}\delta (y-f(x,\xi ))\mathcal{P}\left(\xi \right)\mathrm{d}\xi \hfill \\ \hfill & =\mathcal{P}\left(f_{\xi }^{-1}(x,y)\right){\left|\frac{\partial f}{\partial \xi }(x,f_{\xi }^{-1}(x,y))\right|}^{-1},\hfill \end{array}\end{array}$$

(3)

where we denote with $\mathcal{P}\left(\xi \right|\xi =y-f\left(x\right))$ the distribution of the noise variable $\xi$, with $f_{\xi }^{-1}$ the inverse with respect to the second argument of $f(x,\xi )$, such that $y=f(x,f_{\xi }^{-1}(x,y))$, and with $\delta (·)$ Dirac’s delta distribution. We note that without loss of generality, we can choose the noise variable $\xi$ to be uniformly distributed over its domain ${\mathcal{D}}_{\xi }=[0,\eta ]$ invoking the inverse cumulative distribution function (inverse CDF) method. We can then absorb the inverse CDF transformation into f. In this parameterization, Equation (3) reduces to

$$\begin{array}{c}\hfill p\left(y\right|xX\to Y,\mathcal{M})=\frac{1}{\left|\eta \right|}{\left|\frac{\partial f}{\partial \xi }(x,f_{\xi }^{-1}(x,y))\right|}^{-1}.\end{array}$$

3. From Additive Noise Models to MCM

In this section, we give an overview of the additive noise model (ANM) as a paradigmatic example of a simple asymmetry method and the flexible noise model (FNM) as a more complicated one. We then propose a novel causal model that can overcome some of their limitations.

3.1. The Additive Noise Model

From the very definition of the SCM, it might be reasonable to model the effect of a cause variable X on a variable Y through a deterministic causal mechanism f plus independent noise $\xi$. Such a model is called Additive Noise Model (ANM) [11,23]. In the linear, non-Gaussian, and acyclic models (LiNGAM, see Shimizu et al. [26]), f is linear, and at most one variable between X or $\xi$ is normally distributed. In the nonlinear ANM proposed by Hoyer et al. [11], f is nonlinear and the only constraint on the noise distribution $\mathcal{P}\left(\xi \right)$ is to be independent of the cause X.

Mimicking the approach followed in Section 2.3, we find the ANM’s correspondent conditional density distribution. The generic additive noise model can be written in the form $Y=f\left(X\right)+\xi$; hence,

$$\begin{array}{cc}\hfill p_{\mathrm{ANM}}\left(y\right|x,X\to Y)& ={\int }_{{\mathcal{D}}_{\xi }}p\left(y\right|x,\xi ,X\to Y,\mathrm{ANM})\mathcal{P}\left(\xi \right)\mathrm{d}\xi ={\int }_{{\mathcal{D}}_{\xi }}\delta (y-f\left(x\right)-\xi ))\mathcal{P}\left(\xi \right)\mathrm{d}\xi \hfill \\ \hfill & =\mathcal{P}(y-f(x\left)\right),\hfill \end{array}$$

where $\mathcal{P}\left(\xi \right)$ is the distribution of the noise variable $\xi \in {\mathcal{D}}_{\xi }$. This shows that the conditional probability for an ANM, given a specific causal direction, is the noise distribution evaluated at the residuals coordinates $y-f\left(x\right)$, for the X → Y causal direction. If we furthermore assume the noise to be normally distributed, with a fixed variance ${\sigma }^2$, this further reduces to the Gaussian distribution

$$p_{\mathrm{ANM}}\left(y\right|x,X\to Y)=\mathcal{G}(y-f\left(x\right),{\sigma }^2).$$

(4)

3.2. Flexible Noise Model

Many attempts have been made to go beyond the additive noise formulation (see, e.g., [9,27]) and many algorithms have proven effective, even if they have not been directly related to SCMs. Some of these approaches have identified preferred causal directions by comparing complexity measures of the conditional distributions $p\left(y\right|x$X → Y) and $p\left(x\right|y$Y → X) (see, e.g., Sun et al. [13,22].). Unfortunately, in many cases, the proposed complexity measures are intractable and hard to quantify. In Guardiani et al. [25], we presented a hierarchical generative model that exploits prior information on the observed variables to produce a joint distribution, parametrized as follows

$$p_{\mathrm{FNM}}\left(y\right|x,X\to Y)=e^{g\left(y\right)}\frac{e^{h(x,y)}}{{\int }_{{\mathcal{D}}_x}{e}^{h(\tilde{x},y)}\mathrm{d}\tilde{x}}{(\int {\mathcal{D}}_y\frac{e^{h(x,\tilde{y})+g\left(\tilde{y}\right)}}{{\int }_{{\mathcal{D}}_x}{e}^{h(\tilde{x},\tilde{y})}\mathrm{d}\tilde{x}}\mathrm{d}\tilde{y})}^{-1}.$$

(5)

In the following, we will refer to this model as the flexible noise model (or FNM). In the FNM, the true causal direction is determined by comparing Bayesian evidence (more specifically the lower bounds of the evidence) between different possible causal graphs. As we will discuss in Section 3.7, using conditional–probability model evidences to determine causal directions inherently relates to the complexity of the conditional distribution. Intuitively, the evidence accounts for how many degrees of freedom (read algorithmic complexity) need to be excited in order to reproduce a certain joint distribution given a specific causal direction. The generative nature of Bayesian hierarchical models also allows for an understanding of these models from the SCM perspective, in which Pearl’s do-calculus, interventions, and counterfactuals are naturally defined. If we use Equation (3) for the FNM, we can describe the conditional probability of the X → Y model as

$$\begin{array}{cc}\hfill p_{\mathrm{FNM}}\left(y\right|x,X\to Y)& =e^{g\left(y\right)}\frac{e^{h(x,y)}}{{\int }_{{\mathcal{D}}_x}{e}^{h(\tilde{x},y)}\mathrm{d}\tilde{x}}{\left({\int }_{{\mathcal{D}}_y}\frac{e^{h(x,\tilde{y})+g\left(\tilde{y}\right)}}{{\int }_{{\mathcal{D}}_x}{e}^{h(\tilde{x},\tilde{y})}\mathrm{d}\tilde{x}}\mathrm{d}\tilde{y}\right)}^{-1}\hfill \end{array}$$

(6)

$$\begin{array}{cc}& \stackrel{!}{=}\mathcal{P}\left(f_{\xi }^{-1}(x,y)\right){\left|\frac{\partial f}{\partial \xi }(x,f_{\xi }^{-1}(x,y))\right|}^{-1},\hfill \end{array}$$

(7)

which is subject to the constraint ${\int }_{{\mathcal{D}}_y}{p}_{\mathrm{FNM}}\left(\tilde{y}\right|x,X\to Y)\mathrm{d}\tilde{y}=1$.

While this approach is effective when prior knowledge (regarding the distributions of the observational variables) is accessible, the constraints on f are too weak to guarantee identifiability in a more general setting. In the case that no prior information on the observed variables is available and the conditional distributions are similarly complex in either causal direction, additional asymmetry must be introduced in the model in order to identify the correct causal structure.

3.3. From FNM to First-Order MCM

In the spirit of breaking the symmetry in the split of the joint distribution $p(x,y)=p\left(x\right)p\left(y\right|x)=p(y\left)p\right(x\left|y\right)$, we introduce the moment-constrained causal model (MCM). This model can be seen as a generalization of the nonlinear additive noise model and a (restrictive) modification of the flexible noise model. Starting from the FNM conditional distribution Equation (5), we can represent an x-independent y distribution

$$p_{\mathrm{FNM}}\left(y\right|h(x,y)\equiv 0,X\to Y)=\frac{e^{g\left(y\right)}}{{\int }_{{\mathcal{D}}_y}{e}^{g\left(\tilde{y}\right)}\mathrm{d}\tilde{y}}$$

(8)

by setting $h(x,y)\equiv 0$. Then we realize that the conditional probability of the ANM from Equation (4) can simply be represented from the x-independent probability distribution in Equation (8)

$$p\left(y\right|x,X\to Y)=\frac{e^{g\left(y\right)}}{{\int }_{{\mathcal{D}}_y}{e}^{g\left(\tilde{y}\right)}\mathrm{d}\tilde{y}}\stackrel{!}{=}\mathcal{G}(y-f\left(x\right),{\sigma }^2),$$

(9)

if we substitute $y\mapsto y-f\left(x\right)$ and require that g is quadratic ($g[·]=-\frac{1}{2}{\left(\frac{·}{\sigma }\right)}^2$).

Equation (9) directly shows that a nonlinear ANM can be interpreted as a coordinate transformation that shifts the mean of a normal and zero-centered y-distribution (with fixed variance) by a nonlinear x-dependent function $f\left(x\right)$. We will use this intuition to develop the MCM. We now want to extend the ANM in a way that allows for more complicated noise distributions and noise-cause entanglements, similar to the FNM. As previously discussed though, in the absence of detailed prior knowledge the FNM is not capable of completely constraining the f of its corresponding SCM, nor it is able to impose strong constraints on the shape of the joint distribution $p_{\mathrm{FNM}}\left(x\right|y,$X → Y). In particular, it is not uniquely centered around a smooth function $f\left(x\right)$ as the ANM is.

Mimicking the approach described in Equation (9), we propose to extend the ANM with a model which is parametrized similarly to the FNM, but whose conditional distribution’s first moment is centered around a smooth function of x, which we call $\widehat{s}\left(x\right)$. To build this model, we could start from the FNM defined on some probe coordinates $(x,y^{\prime })$ and perform the change of variable $y=y^{\prime }-\widehat{s}\left(x\right)$. This parameterization though allows for both $h(x,y^{\prime })$ and $\widehat{s}\left(x\right)$ to represent the average ${\langle y^{\prime }\rangle }_{p\left(y^{\prime }\right|x,X\to Y)}$. To eliminate this degeneracy, we introduce a slightly different change of variable

$$y=y^{\prime }-{\langle y^{\prime }\rangle }_{p_{\mathrm{FNM}}\left(y^{\prime }\right|x,X\to Y)}+\widehat{s}\left(x\right).$$

(10)

This way, $\widehat{s}\left(x\right)$ will model the mean of the causal mechanism f and the function $h(x,y^{\prime })$ will model higher-order effects. This parameterization, together with the additional smoothness hypothesis on the functions $\widehat{s}\left(x\right)$ and $h(x,y^{\prime })$ (see Section 3.6) fulfill our initial purpose of further breaking the symmetry in the joint distribution split in the two opposite causal directions X → Y and Y → X.

We will refer to the model in Equation (10) as the first-order MCM. Upon defining $\delta y\left(x\right):=\widehat{s}\left(x\right)-{\langle y^{\prime }\rangle }_{p_{\mathrm{FNM}}\left(y^{\prime }\right|x,X\to Y)}$, we can write the conditional density of the MCM model as

$$\begin{array}{cc}\hfill p_{\mathrm{MCM}}\left(y\right|x,X\to Y)& =e^{g(y^{\prime }-\delta y\left(x\right))}\frac{e^{h(x,y^{\prime }-\delta y\left(x\right))}}{{\int }_{{\mathcal{D}}_x}{e}^{h(\tilde{x},y^{\prime }-\delta y\left(x\right))}\mathrm{d}\tilde{x}}{\mathcal{Z}}^{-1}\left(x\right),\hfill \end{array}$$

(11)

where $y:=y^{\prime }-\delta y\left(x\right)$ and $\mathcal{Z}\left(x\right)$ represents the normalization term. By comparing Equations (9) and (11), we see that the $\delta y\left(x\right)$ in the MCM plays the same role as the $f\left(x\right)$ in the ANM and that all the remaining contributions are modeled by $h(x,y^{\prime }-\delta y\left(x\right))$.

3.4. Second-Order MCM

With the introduction of the first-order MCM, we have shown how to model a conditional distribution centered around a smooth function $\widehat{s}\left(x\right)$ for the causal direction X → Y. The guiding intuition we want to bare in mind is that at least some of the moments of the effect variable, should somehow smoothly depend on the cause. In the first-order MCM, we demonstrate this dependence for the first moment. Taking this concept one step further, we can require that the variance of the MCM conditional density is represented by a smooth, strictly positive, and x-only dependent function $\widehat{\sigma }\left(x\right)$. We do so by modifying the change of variable in Equation (10) into

$$y=\widehat{\sigma }\left(x\right)(\frac{y^{\prime }-\langle y^{\prime }\rangle }{\sqrt{\langle {(y^{\prime }-\langle y^{\prime }\rangle )}^2\rangle }}+\widehat{s}\left(x\right)),$$

(12)

where all expectation values are taken over the FNM conditional distribution ${\langle ·\rangle }_{p_{\mathrm{FNM}}\left(y^{\prime }\right|x,X\to Y)}$. We note that all these expectation values depend on x.

3.5. Likelihood

Without loss of generality, we assume that the observational data d consists of N pairs $(x_i,y_i)$, with $i\in 1,\dots ,N$, which live on some compact space $(x,y)\in {[0,1]}^2$. We bin the data into a fine two-dimensional grid over the x and y directions, such that

$$n_{ij}\left(d\right):=\sum _{m=1}^N{\int }_{i\Delta x}^{(i+1)\Delta x}\mathrm{d}x{\int }_{j\Delta y}^{(j+1)\Delta y}\delta (x-x_m)\delta (y-y_m)\mathrm{d}y$$

is the number of data counts within the ${(i,j)}^{\mathrm{th}}$ where $\Delta x$ and $\Delta y$ are the pixel dimensions in the x and y directions. We then assume a Poisson likelihood

$$\mathcal{P}\left(d\right|\varrho (x,y),X\rightleftarrows Y)=\prod _{i,j}\frac{{\lambda }_{ij}^{n_{ij}}}{n_{ij}!}{e}^{-{\lambda }_{ij}}$$

(13)

to compare the number of counts $n_{ij}$ with the model’s expectations

$${\lambda }_{ij}:={\lambda }_{ij}\left(\varrho (x,y)\right)={\int }_{i\Delta x}^{(i+1)\Delta x}\mathrm{d}x{\int }_{j\Delta y}^{(j+1)\Delta y}\varrho (x,y)\mathrm{d}y\approx \Delta x\Delta y\varrho ((i+\frac{1}{2})\Delta x,(j+\frac{1}{2})\Delta y),$$

(14)

where we indicate with $\varrho (x,y)=\varrho \left(x\right)p\left(y\right|x,$X → Y) (or $\varrho (x,y)=\varrho \left(y\right)p\left(x\right|y,Y\to X)$ depending on the causal model at hand) the joint density of x and y. We note that if the data-generating process is unknown, selection effects and instrument responses can still imprint a wrong causal direction into the data. Even though under certain hypotheses it is still possible to recover the correct causal direction (see Bareinboim et al. and Correa and Bareinboim [28,29]), in general, this is not the case (see, e.g., Section Results in Guardiani et al. [25]). In principle, if we know how the data are measured, we can model the data generation process with an additional and independent likelihood factor ${\mathcal{P}}_{\mathrm{measure}}\left(d_i\right|x_i,y_i,X\rightleftarrows Y)$.

3.6. Inference

In Section 2.2, we explain why the IFT generative model framework is particularly suited to developing causal inference methods. One of the main reasons is that it allows generating and inferring smooth probability distributions from the data d according to a certain causal model. Let us consider the causal direction X → Y. This problem then reduces to learning the joint density ${\varrho }_{\mathrm{MCM}}(x,y|$X → Y$)=\varrho \left(x\right)p_{\mathrm{MCM}}\left(y\right|x,$X → Y) by inferring the correlation structure of the independent density $\varrho \left(x\right)$ parametrized as $\varrho \left(x\right)=e^{r\left(x\right)}$ and of the conditional density $p_{\mathrm{MCM}}\left(y\right|x,$X → Y). Upon denoting all unknown signal parameters with $s:=(r,g,h,\widehat{s})$, the full model reads

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathcal{P}(d,s)& =\mathcal{P}\left(d\right|s\left)\mathcal{P}\right(s),\mathrm{where}\hfill \\ \hfill \mathcal{P}\left(d\right|s)& =\mathcal{P}\left(d\right|\varrho [r,g,h,\widehat{s}])\mathrm{and}\hfill \\ \hfill \mathcal{P}\left(s\right)& =\mathcal{P}\left(r\right|p_r)\mathcal{P}\left(g\right|p_g)\mathcal{P}\left(h\right|p_h)\mathcal{P}\left(\widehat{s}\right|p_{\widehat{s}})\mathcal{P}\left(p_r\right)\mathcal{P}\left(p_g\right)\mathcal{P}\left(p_h\right)\mathcal{P}\left(p_{\widehat{s}}\right).\hfill \end{array}\end{array}$$

(15)

We have denoted with $p_r:=(a_r,k_r,{\gamma }_r)$ (the same holds for g, h, and $\widehat{s}$) the parameters of the Matérn kernel parameterization of the power spectrum   

$$P_r\left(k\right)=\frac{a_r^2}{{\left[1+{(k/k_r)}^2\right]}^{MM{\gamma }_r/2}}.$$

(16)

This is the parameterization we use to infer the correlation structure of the signal fields s for each Fourier mode k. The process of generating smooth distributions with Matérn kernel correlation structures is described in more detail by Guardiani et al. [25].

We then need to infer the signal vector s. We do this by transforming the coordinates of the signal vector $s=s\left(\chi \right)$ such that the prior on the new $\chi$ coordinates becomes an uncorrelated Gaussian $\mathcal{P}\left(\chi \right)=\mathcal{G}(\chi ,𝟙)$ as described by Frank et al. [30]. We note once again that this generative structure resembles the SCM. We then implement the resulting model using the Python package Numerical Information Field Theory (NIFTy [31,32,33]) and finally use NIFTy’s implementation of geometric variational inference (geoVI) [30] to draw posterior samples. In essence, the geoVI scheme finds an approximate coordinate transformation that maps the posterior distribution of the signal parameters $\chi$ into an approximate standard normal distribution. We refer to the bibliography for more details and benchmarks.

3.7. Evidence Estimation

In order to determine the true causal relations underlying the data, we estimate the Bayesian evidence lower bounds (ELBOs) of our models [34,35]. Since we’re using the geoVI variational inference approach described in Section 3.6, we can exploit the variational nature of our minimization scheme to estimate the ELBO. In fact, the ELBO can be related to the Kullback–Leibler divergence ${\mathcal{D}}_{\mathrm{KL}}$ between the approximating distribution $\mathcal{Q}$ to the true posterior $p\left(s\right(\chi \left)\right|d)$

$$ln\left(p\left(d\right)\right)-{\mathcal{D}}_{\mathrm{KL}}\left(\mathcal{Q}\left(s\left(\chi \right)\right|d)\right|p\left(s\left(\chi \right)\right|d))=-{\langle \mathcal{H}(s\left(\chi \right),d)\rangle }_{\mathcal{Q}\left(s\right|d)}+\frac{1}{2}\left(N+\mathrm{Tr}log{\Lambda }_{\Theta }{|}_{\chi =\overline{\chi }}\right),$$

(17)

where, as before, $s\left(\chi \right)$ represents the parameterization of model degrees of freedom (signal), $d={\{x_i,y_i\}}_{i=1\dots N}$ is the data, and ${\Lambda }_{{\Theta |}_{\chi =\overline{\chi }}}$ represents the diagonalized metric of the posterior at the current sampling point. We have also denoted with $\mathcal{H}(·):=-logp(·)$ the information Hamiltonian. The expectation value ${\langle \mathcal{H}(s\left(\chi \right),d)\rangle }_{\mathcal{Q}}$ is estimated via geoVI posterior sample average. The ${\mathcal{D}}_{\mathrm{KL}}$ represents the Kullback–Leibler divergence between the approximating $\mathcal{Q}\left(s\right(\chi \left)\right|d)$ distribution and the posterior $p\left(s\right(\chi \left)\right|d)$. Given that ${\mathcal{D}}_{\mathrm{KL}}\ge 0$, it is more convenient to consider only the lower bound

$$ln\left(p\left(d\right)\right)\ge -{\langle \mathcal{H}(s\left(\chi \right),d)\rangle }_{\mathcal{Q}\left(s\right|d)}+\frac{1}{2}\left(N+\mathrm{Tr}log{\Lambda }_{\Theta }{|}_{\chi =\overline{\chi }}\right).$$

(18)

The causal direction with the highest ELBO on a given dataset d is our final estimated causal direction. We note that, for well-constructed models, the ELBO can quantify the complexity of the conditional probability distribution of the effect given the cause, since it inherently penalizes the excitation of model degrees of freedom (Occam’s razor principle).

3.8. Implementation

We implemented a preliminary version of the first-order MCM in Python and NIFTy. This implementation makes use of the inference machinery described in Section 3.6 to produce posterior reconstructions of the data d according to the MCM model.

An example of the MCM performance is shown in Figure 1. Even though it has so far only been tested on the bci_default dataset described by Kurthen et al. [16], from the reconstruction of the conditional density ${\rho }_{\mathrm{FNM}}(x,y)$ (Figure 1, middle column) it can be already seen that the symmetry between cause and effect is clearly broken. In the true causal direction, the $\widehat{s}$ field can capture more x-dependent structures from the data. For this case, the ELBO calculations support the correct X → Y causal-direction model by a Bayesian factor of ≈$e^{20}$.

4. Conclusions

In this work, we showed the connection between SCMs and Bayesian generative modeling within the framework provided by IFT. In this spirit, we introduced a novel causal model that translates the intuition that some of the moments of the effect variable should exhibit a smooth dependence on the cause into an algorithm that explicitly breaks the symmetry in the split of the joint distribution of cause and effect. Preliminary results on a single dataset show that our new model can identify the correct causal direction. Future work directions include benchmarking first- and second-order MCM against the state-of-the-art causal inference algorithms as well as exploring possible higher-order correction terms.