Automating Model Comparison in Factor Graphs

Abstract

Bayesian state and parameter estimation are automated effectively in a variety of probabilistic programming languages. The process of model comparison on the other hand, which still requires error-prone and time-consuming manual derivations, is often overlooked despite its importance. This paper efficiently automates Bayesian model averaging, selection, and combination by message passing on a Forney-style factor graph with a custom mixture node. Parameter and state inference, and model comparison can then be executed simultaneously using message passing with scale factors. This approach shortens the model design cycle and allows for the straightforward extension to hierarchical and temporal model priors to accommodate for modeling complicated time-varying processes.

1. Introduction

The famous aphorism of George Box states that “all models are wrong, but some are useful” [1]. It is the task of statisticians and data analysts to find a model which is most useful for a given problem. The build, compute, critique and repeat cycle [2], also known as Box’s loop [3], is an iterative approach for finding the most useful model. Any efforts in shortening this design cycle increase the chances of developing more useful models, which in turn might yield more reliable predictions, more profitable returns or more efficient operations for the problem at hand.

In this paper, we choose to adopt the Bayesian formalism, and therefore we specify all tasks in Box’s loop as principled probabilistic inference tasks. In addition to the well-known parameter and state inference tasks, the critique step in the design cycle is also phrased as an inference task, known as Bayesian model comparison, which automatically embodies Occam’s razor (Ch. 28.1, [4]). As opposed to just selecting a single model in the critique step, for different models, we better quantify our confidence about which model is best, especially when data are limited (Ch. 18.5.1, [5]). The uncertainty arising from prior beliefs $p\left(m\right)$ over a set of models m and limited observations can be naturally included through the use of Bayes’ theorem:

$$p\left(m|D\right)=\frac{p\left(D|m\right)p\left(m\right)}{p\left(D\right)},$$

(1)

which describes the posterior probabilities $p\left(m|D\right)$ as a function of model evidences $p\left(D|m\right)$ and where $p\left(D\right)={\sum }_{m}p\left(D|m\right)p\left(m\right)$. Starting from Bayes’ rule, we can obtain different comparison methods from the literature, such as Bayesian model averaging [6], selection, and combination [7], which we formally introduce in Section 5. We use Bayesian model comparison as an umbrella term for these three methods throughout this paper.

The task of state and parameter estimation was automated in a variety of tools, e.g., [8,9,10,11,12,13,14]. Bayesian model comparison, however, is often regarded as a separate task, whereas it submits to the same Bayesian formalism as state and parameter estimation. A reason for overlooking the model comparison stage in a modeling task is that the computation of model evidence $p\left(D|m\right)$ is, in most cases, not automated and therefore still requires error-prone and time-consuming manual derivations, in spite of its importance and the potential data representation improvement that can be achieved by, for example, including a Bayesian model combination stage in the modeling process [7].

This paper aims to automate the Bayesian model comparison task and is positioned between the mixture model and ‘gates’ approaches of [15,16], respectively, which we will describe in more detail in Section 2. Specifically, we specify the model comparison tasks as a mixture model, similarly to that in [15], on a factor graph with a custom mixture node for which we derive automatable message-passing update rules, which perform both parameter and state estimation, and model comparison. These update rules generalize the model comparison to arbitrary models submitting to exact inference, as the operations of the mixture node are ignorant about the adjacent subgraphs. Additionally, we derive three common model comparison methods from the literature (Bayesian model averaging, selection, and combination) using the custom mixture node.

In short, this paper derives automated Bayesian model comparison using message passing in a factor graph. After positioning our paper in Section 2 and after reviewing factor graphs and message passing-based probabilistic inference in Section 3, we make the following contributions:

• We show that the Bayesian model comparison can be performed through message passing on a graph, where the individual model performance results are captured in a single factor node as described in Section 4.1.
• We specify a universal mixture node and derive a set of custom message-passing update rules in Section 4.2. Performing probabilistic inference with this node in conjunction with scale factors yields different Bayesian model comparison methods.
• Bayesian model averaging, selection, and combination are recovered and consequently automated in Section 5.1, Section 5.2 and Section 5.3 by imposing a specific structure or local constraints on the model selection variable m.

We verify our presented approach in Section 6.1. We illustrate its use for models with both continuous and discrete random variables in Section 6.2.1, after which we continue with an example of voice activity detection in Section 6.2.2, where we add temporal structure on m. Section 7 discusses our approach, and Section 8 concludes the paper.

2. Related Work

This section discusses related work and aims at providing the clear positioning of this paper for our contributions that follow in the upcoming sections.

The task of model comparison is widely represented in the literature [17], for example, concerning hypothesis testing [18,19]. Bayesian model averaging [6] can be interpreted as the simplest form of model comparison that uses the Bayesian formalism to retain the first level of uncertainty in the model selection process [20]. Bayesian model averaging has proven to be an effective and principled approach that converges with infinite data to the single best model in the set of candidate models [21,22,23]. When the true underlying model is not part of this set, the data is often better represented by ad hoc methods [24], such as ensemble methods. In [7], the idea of Bayesian model comparison is introduced, which basically performs Bayesian model averaging between mixture models comprising the candidate models with different weights. Another ensemble method is proposed in [23,25], which uses (hierarchical) stacking [26] to construct predictive densities whose weights are data dependent.

Automating the model design cycle [2] under the Bayesian formalism has been the goal of many probabilistic programming languages [8,9,10,11,12,13,14]. This paper focuses on message passing-based approaches, which leverage the conditional independencies in the model structure for performing probabilistic inference, e.g., [27,28,29,30], which will be formally introduced in Section 3.2. Contrary to alternative sampling-based approaches, message passing excels in modularity, speed and efficiency, especially when models submit to closed-form (variational) message computations. Throughout this paper, we follow the spirit of [31], which shows that many probabilistic inference algorithms, such as (loopy) belief propagation [32,33], variational message passing [30,34], expectation maximization [35], and expectation propagation [36] can all be phrased as a constrained Bethe free energy [37] minimization procedure. Specifically, in Section 5, we aim to phrase different Bayesian model comparison methods as automatable message-passing algorithms. Not only does this have the potential to shorten the design cycle but also to develop novel model comparison schemes.

The connection between (automatable) state and parameter inference, versus model comparison was explored recently by [15,22], who frame the problem of model comparison as a “mixture model estimation” task that is obtained by combining the individual models into a mixture model with weights representing the model selection variable. The exposition in [15,22] is based on relatively simple examples that do not easily generalize to more complex models for the model selection variable and for the individual cluster components. In the current paper, we aim to generalize the mixture model estimation approach by an automatable message passing-based inference framework. Specifically, we build on the results of the recently developed scale factors (Ch. 6, [38,39]), which we introduce in Section 3.3. These scale factors support the efficient tracking of local summaries of model evidences, thus enabling model comparison in generic mixture models; see Section 4 and Section 5.

The approach we present in the current paper is also similar to the concept of ‘gates’, introduced in [16]. Gates are factor nodes that switch between mixture components with which we can derive automatable message-passing procedures. Mathematically, a gate represents a factor node of the form $f(s,m)={\prod }_{k=1}^K{f}_k{\left(s_k\right)}^{m_k}$, where the model selection variable m is a one-of-K coded vector defined as $m_k\in \{0,1\}$ subject to ${\sum }_{k=1}^K{m}_k=1$. The variables are $s={\bigcup }_{k=1}^K{s}_k$. Despite the universality of the Gates approach, the inference procedures in [16] focus on variational inference [30,34,40] and expectation propagation [36]. The mixture selection variable m is then updated based on “evidence-like quantities”. In the variational message-passing case, these quantities resemble local Bethe free energy contributions, which only take into account the performance around the gate factor node, disregarding the performance contributions of other parts in the model. Because of the local contributions, the message-passing algorithm can be very sensitive to its initialization, which has the potential to yield suboptimal inference results.

In the current paper, we extend gates to models submitting to exact inference using scale factors, which allows for generalizing and automating the mixture models of [15,22]. With these advances we can automate well-known Bayesian model comparison methods using message passing, enabling the development of novel comparison methods.

3. Background Material

This section aims to provide a concise review of factor graphs and message-passing algorithms, as we deem these concepts essential to appreciate our core contributions, which we present in Section 4 and Section 5. This review is intentionally not exhaustive; instead, we provide references to works that help to obtain a deeper understanding about the material covered here. In Section 3.1, we introduce factor graphs as a way to visualize factorizable (probabilistic) models. Section 3.2 then describes how probabilistic inference can be efficiently performed through message passing, utilizing the inherent factorization of the model. The model evidence can be tracked locally with message passing, using scale factors as described in Section 3.3. Finally, Section 3.4 introduces the variational free energy as a bound on the model evidence.

3.1. Forney-Style Factor Graphs

A factor graph is a specific type of probabilistic graphical model. Here we use the Forney-style factor graph (FFG) framework as introduced in [41] with notational conventions adopted from [27] to visualize our probabilistic models. An FFG can be used to represent a factorized function

$$f\left(s\right)=\prod _{a\in \mathcal{V}}{f}_a\left(s_a\right),$$

(2)

where s collects all variables in the function. The subset $s_a\subseteq s$ contains all argument variables of a single factor $f_a$. FFGs visualize the factorization of such a function as a graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where nodes $\mathcal{V}$ and edges $\mathcal{E}\subseteq \mathcal{V}\times \mathcal{V}$ represent factors and variables, respectively. An edge connects to a node only if the variable associated with the edge is an argument of the factor associated with the node. Nodes are indexed by the variables a, b, and c, where edges are indexed by i and j unless stated otherwise. The set of edges connected to node $a\in \mathcal{V}$ is denoted by $\mathcal{E}\left(a\right)$, and the set of nodes connected to edge $i\in \mathcal{E}$ is referred to as $\mathcal{V}\left(i\right)$. As an example, consider the model $f(s_1,s_2,s_3,s_4)$ with factorization

$$f(s_1,s_2,s_3,s_4)=f_a\left(s_1\right)f_b(s_1,s_2)f_c\left(s_3\right)f_d(s_2,s_3,s_4)$$

(3)

The FFG representation of (3) is shown in Figure 1. For a more thorough review of factor graphs, we refer the interested reader to [27,28].

3.2. Sum-Product Message Passing

Consider the normalized probabilistic model

$$p(y,s)=\prod _{a\in \mathcal{V}}{f}_a(y_a,s_a),$$

(4)

with observed and latent sets of variables y and s, respectively. Note here that the subset $y_a\subseteq y$ can be empty, for example, when dealing with prior distributions. Upon observing the realizations $\widehat{y}$, the corresponding model $p(y=\widehat{y},s)$ becomes unnormalized. Probabilistic inference in this model then concerns the computation of the posterior distribution over the latent variables $p(s|y=\widehat{y})$ and of the model evidence $p(y=\widehat{y})$ as $p(y=\widehat{y},s)=p(s|y=\widehat{y})p(y=\widehat{y})$. Consider the global integration over all variables in (4), except for $s_j$ as $\int p(y=\widehat{y},s)\mathrm{d}{s}_{\setminus j}$. (Integrals are taken over the support over the variables. If a variable is discrete valued, integral operators are replaced with summation operators. For a consistent exposition of our work, we use integral operators throughout the paper.)  This large global integration can be performed through a set of smaller local computations as a result of the assumed factorization in (4). These smaller local computations can be considered to summarize the part of the graph that is being integrated over and are termed messages, which graphically can be interpreted to propagate over the edges in the graph. These messages are denoted by $\mu$ and can be locally computed on the graph. The sum–product message ${\overrightarrow{\mu }}_{s_j}\left(s_j\right)$ flowing out of the node $f_a(y_a={\widehat{y}}_a,s_a)$ with incoming messages ${\overrightarrow{\mu }}_{s_i}\left(s_i\right)$ is given by [29]

$${\overrightarrow{\mu }}_{s_j}\left(s_j\right)=\int f_a(y_a={\widehat{y}}_a,s_a)\prod _{i\ne j}{\overrightarrow{\mu }}_{s_i}\left(s_i\right)\mathrm{d}{s}_{a\setminus j}.$$

(5)

We represent edges in the graph by arbitrarily directed arrows in order to distinguish between forward and backward messages propagating in or against the direction of an edge $s_j$ as ${\overrightarrow{\mu }}_{s_j}\left(s_j\right)$ and ${\overrightarrow{\mu }}_{s_j}\left(s_j\right)$, respectively. Following this approach, the global integration reduces to the product of the messages of the variable of interest as $\int p(y=\widehat{y},s)\mathrm{d}{s}_{\setminus j}={\overrightarrow{\mu }}_{s_j}\left(s_j\right){\overrightarrow{\mu }}_{s_j}\left(s_j\right)$ for acyclic models.

Posterior distributions on edges and around nodes can then be computed according to

$$p(s_j|y=\widehat{y})=\frac{{\overrightarrow{\mu }}_{s_j}\left(s_j\right){\overrightarrow{\mu }}_{s_j}\left(s_j\right)}{{\displaystyle \int {\overrightarrow{\mu }}_{s_j}\left(s_j\right){{\displaystyle \overrightarrow{{\displaystyle \mu }}}}_{s_j}\left(s_j\right)\mathrm{d}{s}_j}}$$

(6)

and

$$p(s_a|y=\widehat{y})=\frac{{\displaystyle f_a(y_a={\widehat{y}}_a,s_a)\prod _{i\in \mathcal{E}\left(a\right)}{\overrightarrow{\mu }}_{s_i}\left(s_i\right)}}{{\displaystyle \int f_a(y_a={\widehat{y}}_a,s_a)\prod _{i\in \mathcal{E}\left(a\right)}{\overrightarrow{\mu }}_{s_i}\left(s_i\right)\mathrm{d}{s}_a}},$$

(7)

respectively [31].

Derivations of the message-passing update rule in (5) by phrasing inference as a constrained Bethe free energy minimization procedure are presented in [31]. Through a similar procedure, one can obtain alternative message-passing algorithms, such as variational message passing [30,34,40], expectation propagation [36], expectation maximization [35] and hybrid algorithms.

3.3. Scale Factors

The previously discussed integration $\int p(y=\widehat{y},s)\mathrm{d}{s}_{\setminus j}$ can be represented differently as

$$\int p(y=\widehat{y},s)\mathrm{d}{s}_{\setminus j}=p(y=\widehat{y})\int p(s|y=\widehat{y})\mathrm{d}{s}_{\setminus j}=p(y=\widehat{y})p(s_j|y=\widehat{y}),$$

(8)

where $p(s_j|y=\widehat{y})$ is the marginal distribution of $s_j$. The implications of this result are significant: the product of two colliding sum–product messages ${\overrightarrow{\mu }}_{s_j}\left(s_j\right){\overrightarrow{\mu }}_{s_j}\left(s_j\right)$ in an acyclic graph results in the scaled marginal distribution $p(y=\widehat{y})p(s_j|y=\widehat{y})$. Because of the normalization property of $p(s_j|y=\widehat{y})$, it is possible to obtain both the normalized posterior $p(s_j|y=\widehat{y})$ as the model evidence $p(y=\widehat{y})$ on any edge and around any node in the graph.

Theorem 1.

Consider an acyclic Forney-style factor graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$. The model evidence of the corresponding model $p(y=\widehat{y},s)$ can be computed at any edge in the graph as $\int {\overrightarrow{\mu }}_{s_j}\left(s_j\right){\overrightarrow{\mu }}_{s_j}\left(s_j\right)\mathrm{d}{s}_j$ for all $j\in \mathcal{E}$ and at any node in the graph as $\int f_a(y_a={\widehat{y}}_a,s_a){\prod }_{i\in \mathcal{E}\left(a\right)}{\overrightarrow{\mu }}_{s_i}\left(s_i\right)\mathrm{d}{s}_a$ for all $a\in \mathcal{V}$.

Proof. 

See Appendix A.1.    □

What enables this local computation of the model evidence is the scaling of the messages resulting from the equality in (5). As a result, the messages ${\overrightarrow{\mu }}_{s_j}\left(s_j\right)$ can be decomposed as

$${\overrightarrow{\mu }}_{s_j}\left(s_j\right)={\overrightarrow{\beta }}_{s_j}{\overrightarrow{p}}_{s_j}\left(s_j\right),$$

(9)

where ${\overrightarrow{p}}_{s_j}\left(s_j\right)$ denotes the probability distribution representing the normalized functional form of the message ${\overrightarrow{\mu }}_{s_j}\left(s_j\right)$. The term ${\overrightarrow{\beta }}_{s_j}$ denotes the scaling of the message ${\overrightarrow{\mu }}_{s_j}\left(s_j\right)$, also known as the scale factor (Ch. 6, [38,39]). Scale factors can be interpreted as local summaries of the model evidence that are passed along the graph.

3.4. Variational Free Energy

In practice, however, the computation of the model evidence and, therefore, the posterior distribution is intractable. Variational inference provides a generalized view that supports probabilistic inference in these types of models by approximating the exact posterior $p(s|y=\widehat{y})$ with a variational posterior $q\left(s\right)$ that is constrained to be within a family of distributions $q\in \mathcal{Q}$. Variational inference optimizes (the parameters of) the variational distribution $q\left(s\right)$ by minimizing the variational free energy (VFE) of a single model, defined as

$$\mathrm{F}\left[q\right]={\mathbb{E}}_{q\left(s\right)}\left[ln\frac{q\left(s\right)}{p(y=\widehat{y},s)}\right]=\mathrm{KL}\left[q\left(s\right)\parallel p(s|y=\widehat{y})\right]-lnp(y=\widehat{y}),$$

(10)

through, for example, coordinate or stochastic gradient descent.

The variational free energy can serve as a bound to the model evidence in (1) for model comparison (Ch. 10.1.4, [42,43,44]). It is important to emphasize that the VFE not only encompasses the model evidence but also the Kullback–Leibler ($\mathrm{KL}$) divergence between the variational and exact posterior distributions obtained from the inference procedure.

4. Universal Mixture Modeling

This section derives a custom factor node that allows for performing model comparison as an automatable message-passing procedure in Section 5. In Section 4.1, we specify a variational optimization objective for multiple models at once, where the optimization of the model selection variable can be rephrased as a probabilistic inference procedure on a separate graph with a factor node encapsulating the model-specific performance metrics. Section 4.2 further specifies this node and derives custom message-passing update rules that allow for jointly computing (1) and for performing state and parameter inference.

Before continuing our discussion, let us first describe the notational conventions adopted throughout this section. In Section 3, only a single model is considered. Here, we cover K normalized models, selected by the model selection variable m, which comprises a 1-of-K binary vector with elements $m_k\in \{0,1\}$ constrained by ${\sum }_{k=1}^K{m}_k=1$. The individual models $p(y_k,s_k|m_k=1)$ are indexed by k, where $y_k$ and $s_k$ collect the observed and latent variables in that model.

4.1. A Variational Free Energy Decomposition for Mixture Models

Consider the normalized joint model

$$p(y,s,m)=p\left(m\right)\prod _{k=1}^{K}p{(y_k,s_k|m_k=1)}^{m_k}$$

(11)

specifying a mixture model over the individual models $p(y_k,s_k|m_k=1)$, with a prior $p\left(m\right)$ on the model selection variable m and where $y={\bigcup }_{k=1}^K{y}_k$ and $s={\bigcup }_{k=1}^K{s}_k$. Based on this joint model, let us define its variational free energy $\mathrm{F}$ as

$$\begin{array}{cc}\hfill \mathrm{F}\left[q\right]& ={\mathbb{E}}_{q(s,m)}\left[ln\frac{q(s,m)}{p(y=\widehat{y},s,m)}\right],\hfill \\ \hfill & ={\mathbb{E}}_{q\left(m\right)}\left[ln\frac{q\left(m\right)}{p\left(m\right)}\right]+{\mathbb{E}}_{q\left(m\right)}\left[\prod _{k=1}^K{\left({\mathrm{F}}_k\left[q\right]\right)}^{m_k}\right],\hfill \end{array}$$

(12)

in which the joint variational posterior $q(s,m)$ factorizes as $q(s,m)=q\left(m\right){\prod }_{k=1}^{K}q{(s_k\mid m_k=1)}^{m_k}$, and where ${\mathrm{F}}_k$ denotes the variational free energy of the k-th model. This decomposition is obtained by noting that m is a 1-of-K binary vector. Furthermore, derivations of this decomposition are provided in Appendix B.1.

This definition also appeared in a similar form in (Ch.10.1.4, [42]) and in the reinforcement learning and active inference community as an approach to policy selection (Section 2.1, [45]). From this definition, it can be noted that the VFE for mixture models can also be written as

$$\mathrm{F}\left[q\right]={\mathbb{E}}_{q\left(m\right)}\left[ln\frac{q\left(m\right)}{p\left(m\right)f_m\left(m\right)}\right]$$

(13)

where

$$f_m\left(m\right)=\prod _{k=1}^{K}exp{(-{\mathrm{F}}_k\left[q\right])}^{m_k},$$

(14)

as shown in Appendix B.1. This observation implies that the obtained VFEs of the individual submodels can be combined into a single factor node $f_m$, representing a scaled categorical distribution, which terminates the edge corresponding to m as shown in Figure 2. The specification of $f_m$ allows for performing inference in the overcoupling model following existing inference procedures, similar to that for the individual submodels. This is in line with the validity of Bayes’ theorem in (1) for both state and parameter inference, and model comparison. Importantly, the computation of the VFE in acyclic models can be automated [31]. Therefore, model comparison itself can also be automated. For cyclic models, one can resort to approximating the VFE with the Bethe free energy [31,37].

In practice, the prior model $p\left(m\right)$ might have hierarchical or temporal dynamics, including additional latent variables. These can be incorporated without loss of generality due to the factorizable structure of the joint model, supported by the modularity of factor graphs and the corresponding message-passing algorithms, as shown in Figure 2.

4.2. A Factor Graph Approach to Universal Mixture Modeling: A General Recipe

In this subsection, we present the general recipe for computing the posterior distributions over the variables s and model selection variable m in universal mixture models. Section 4.3 provides an illustrative example that aids the exposition in this section. The order in which these two section are read is a matter of personal preference.

In many practical applications, distinct models $p(y_k,s_k|m_k=1)$ partly overlap in both structure and variables. These models may, for example, just differ in terms of priors or likelihood functions. Let $f_o(y_o,s_o)$ be the product of factors which are present in all different factorizable models $p(y_k,s_k|m_k=1)$, with overlapping variables $s_o={\bigcap }_{k=1}^K{s}_k$ and $y_o={\bigcap }_{k=1}^K{y}_k$. Based on this description, we define a universal mixture model as in [15] encompassing all individual models as

$$\begin{array}{cc}\hfill p(y,s,m)& =p\left(m\right)\prod _{k=1}^{K}p{(y_k,s_k|m_k=1)}^{m_k}\hfill \\ \hfill & =p\left(m\right)f_o(y_o,s_o)\prod _{k=1}^K{\left(\frac{p(y_k,s_k|m_k=1)}{f_o(y_o,s_o)}\right)}^{m_k},\hfill \end{array}$$

(15)

with model selection variable m. Here, the overlapping factors are factored out from the mixture components. Figure 3 shows a visualization of the transformation from K distinct models into a single mixture model. With the transformation from the different models into a single mixture model presented in Figure 3, it becomes possible to include the model selection variable m into the same probabilistic model.

In these universal mixture models, we are often interested in computing the posterior distributions of (1) the overlapping variables $s_o$ marginalized over the distinct models and of (2) the model selection variable m. Given the posterior distributions $q\left(s_o|m_k=1\right)$ over variables $s_o$ in a single model $m_k$, we can compute the joint posterior distribution over all overlapping variables $q\left(s_o\right)$ as

$$q\left(s_o\right)={\mathbb{E}}_{q\left(m\right)}\left[\prod _{k=1}^{K}q{\left(s_o|m_k=1\right)}^{m_k}\right],$$

(16)

marginalized over the different models m. In the generic case, this computation follows a three-step procedure. First, the posterior distributions $q\left(s_k|m_k=1\right)$ are computed in the individual submodels through an inference algorithm of choice. Then, based on the computed VFE ${\mathrm{F}}_k\left[q\right]$ of the individual models, the variational posterior $q\left(m\right)$ can be calculated. Finally, the joint posterior distribution $q\left(s_o\right)$ can be computed using (16).

Here, we restrict ourselves to acyclic submodels. We show that the previously described inference procedure for computing the joint posterior distributions can be performed jointly with the process of model comparison through message passing with scale factors. In order to arrive at this point, we combine the different models into a single mixture model inspired by [15,22]. Our specification of the mixture model, however, is more general compared to that of [15,22], as it does not constrain the hierarchical depth of the overlapping or distinct models and also works for nested mixture models.

Table 1. Table containing (top) the Forney-style factor graph representation of the mixture node. (bottom) The derived outgoing messages for the mixture node. It can be noted that the backward message towards m resembles a scaled categorical distribution and that the forward message towards $s_j$ represents a mixture distribution. Derivations of the messages ${\overrightarrow{\mu }}_m\left(m\right)$ and ${\overrightarrow{\mu }}_{s_j}\left(s_j\right)$ are presented in Appendices Appendix B.2 and Appendix B.3, respectively.

Factor Node
Messages	Functional form
${\overrightarrow{\mu }}_m\left(m\right)$	${\displaystyle \prod _{k=1}^K{\left(\int {\overrightarrow{\mu }}_{s_j|m_k=1}\left(s_j\right){{\displaystyle \overrightarrow{{\displaystyle \mu }}}}_{s_j}\left(s_j\right)\mathrm{d}{s}_j\right)}^{m_k}}$
${\overrightarrow{\mu }}_{s_j}\left(s_j\right)$	${\displaystyle \sum _{k=1}^K{\overrightarrow{\mu }}_m\left(m_k=1\right){\overrightarrow{\mu }}_{s_j|m_k=1}\left(s_j\right)}$
${\overrightarrow{\mu }}_{s_j|m_k=1}\left(s_j\right)$	${\displaystyle {{\displaystyle \overrightarrow{{\displaystyle \mu }}}}_{s_j}\left(s_j\right)}$

introduces the novel mixture node, which acts as a switching mechanism between different models, based on the selection variable m. It connects the model selection variable m and the overlapping variables $s_j|m_k=1$ for the different models $m_k$, to the variable $s_j$ marginalized over m. Here, the variables $s_j$ connect the overlapping to the non-overlapping factors.

The messages in Table 1 are derived in Appendices Appendix B.2 and Appendix B.3, and can be intuitively understood as follows. The message ${\overrightarrow{\mu }}_m\left(m\right)$ represents the unnormalized distribution over the model evidence corresponding to the individual models. Based on the scale factors of the incoming messages, the model evidence can be computed. The message ${\overrightarrow{\mu }}_{s_j|m_k=1}\left(s_j\right)$ equals the incoming message from the likelihood ${\overrightarrow{\mu }}_{s_j}\left(s_j\right)$. It will update $s_j|m_k=1$ as if the k-th model is active. The message ${\overrightarrow{\mu }}_{s_j}\left(s_j\right)$ represents a mixture distribution over the incoming messages ${\overrightarrow{\mu }}_{s_j}\left(s_j\right)$, where the weightings are determined by the message ${\overrightarrow{\mu }}_m\left(m\right)$ and the scale factors of the messages ${\overrightarrow{\mu }}_{s_j|m_k=1}\left(s_j\right)$. This message can be propagated as a regular message over the overlapping model segment yielding the marginal posterior distributions over all variables in the overlapping model segment.

Theorem 2.

Consider multiple acyclic FFGs. Given the message ${\overrightarrow{\mu }}_{s_j}\left(s_j\right)$ in Table 1 that is marginalized over the different m models, propagating this message through the factor $f_a(y_a,s_a)$, which overlaps for all models with $s_j\in s_a$, yields again messages which are marginalized over the different models.

Proof. 

See Appendix A.2.    □

4.3. A Factor Graph Approach to Universal Mixture Modeling: An Illustrative Example

Consider the two probabilistic models

$$\begin{array}{cc}\hfill p(y,s|m_1=1)& =p\left(y|s\right)p(s|m_1=1),\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill p(y,s|m_2=1)& =p\left(y|s\right)p(s|m_2=1),\hfill \end{array}$$

(18)

which share the same likelihood model with a single observed and latent variable y and s, respectively. The model selection variable m is subject to the prior

$$p\left(m\right)=\mathrm{Ber}\left(m|\pi \right)={\pi }^{m_1}{(1-\pi )}^{m_2},$$

(19)

with $\pi$ denoting the success probability. This allows for the specification of the mixture model

$$p(y,s,m)=p\left(m\right)p\left(y|s\right)\prod _{k=1}^{2}p{\left(s|m_k=1\right)}^{m_k},$$

(20)

which we visualize in Figure 4.

Suppose we are interested in computing the posterior probabilities $p(s|y=\widehat{y})$, marginalized over the distinct models, and $p(m|y=\widehat{y})$. The model evidence of both models can be computed using scale factors locally on the edge corresponding to s as

$$\begin{array}{cc}\hfill p(y=\widehat{y}|m_1=1)& =\int {\overrightarrow{\mu }}_{s|m_1=1}\left(s\right){{\displaystyle \overrightarrow{{\displaystyle \mu }}}}_s\left(s\right)\mathrm{d}s=\int p\left(s|m_1=1\right)p(y=\widehat{y}|s)\mathrm{d}s,\hfill \\ \hfill p(y=\widehat{y}|m_2=1)& =\int {\overrightarrow{\mu }}_{s|m_2=1}\left(s\right){{\displaystyle \overrightarrow{{\displaystyle \mu }}}}_s\left(s\right)\mathrm{d}s=\int p\left(s|m_2=1\right)p(y=\widehat{y}|s)\mathrm{d}s,\hfill \end{array}$$

which takes place inside the mixture node for computing ${\overrightarrow{\mu }}_m\left(m\right)$. Together with the forward message over edge m, we obtain the posterior

$$p(m|y=\widehat{y})=\frac{{\overrightarrow{\mu }}_m\left(m\right){\overrightarrow{\mu }}_m\left(m\right)}{{\sum }_{k=1}^2{\overrightarrow{\mu }}_m\left(m_k=1\right){\overrightarrow{\mu }}_m\left(m_k=1\right)}=\frac{p\left(m\right)p(y=\widehat{y}|m)}{p(y=\widehat{y})}.$$

(21)

The posterior distribution over s for the first model can be computed as

$$p(s|y=\widehat{y},m_1=1)=\frac{{\overrightarrow{\mu }}_{s|m_1=1}\left(s\right){\overrightarrow{\mu }}_s\left(s\right)}{p(y=\widehat{y}|m_1=1)}.$$

From (16), we can then compute the posterior distribution over s marginalized over both models as

$$\begin{array}{cc}\hfill p(s|y=\widehat{y})& =p(m_1=1|y=\widehat{y})p(s|y=\widehat{y},m_1=1)\hfill \\ \hfill & +p(m_2=1|y=\widehat{y})p(s|y=\widehat{y},m_2=1),\hfill \\ \hfill & =\frac{p\left(m_1=1\right)p(y=\widehat{y}|m_1=1)}{p(y=\widehat{y})}\frac{{\overrightarrow{\mu }}_{s|m_1=1}\left(s\right){\overrightarrow{\mu }}_s\left(s\right)}{p(y=\widehat{y}|m_1=1)}\hfill \\ \hfill & +\frac{p\left(m_2=1\right)p(y=\widehat{y}|m_2=1)}{p(y=\widehat{y})}\frac{{\overrightarrow{\mu }}_{s|m_2=1}\left(s\right){\overrightarrow{\mu }}_s\left(s\right)}{p(y=\widehat{y}|m_2=1)},\hfill \\ \hfill & =\frac{\left(p\left(m_1=1\right){\overrightarrow{\mu }}_{s|m_1=1}\left(s\right)+p\left(m_2=1\right){\overrightarrow{\mu }}_{s|m_2=1}\left(s\right)\right){\overrightarrow{\mu }}_s\left(s\right)}{p(y=\widehat{y})}.\hfill \end{array}$$

5. Model Comparison Methods

In this section, we introduce three Bayesian model comparison methods from the literature: model averaging [6], selection and combination [7]. For each of these methods, we describe how to automate them using message passing with the mixture node in Table 1. The factor graph approach here aids the intuitive understanding of the different approaches, as their distinctions are sometimes obscure in the literature. As we will show, each method describes an inference procedure on a slightly different model for the model selection variable m, possibly with different variational constraints as visualized in Figure 5.

5.1. Bayesian Model Averaging

Bayesian model averaging (BMA) can be considered the simplest form of model comparison and is therefore mentioned in many works, e.g., (Ch. 14.1, [6,42]). BMA completes the model specification by specifying a categorical prior distribution over the models m as

$$p\left(m\right)=\mathrm{Cat}\left(m|\pi \right),$$

(22)

where $\pi$ denotes the vector of event probabilities. BMA then aims at computing the posterior distribution over the models $q\left(m\right)$. Given a set of possible models, or hypotheses, with BMA, the posterior distribution $q\left(m\right)$ converges with infinite data to a Kronecker delta function that selects the single model which is the most likely given the observed set of data [15,21]. Figure 5a provides a visual representation of Bayesian model averaging.

5.2. Bayesian Model Selection

Bayesian model selection (BMS) is a further specification of BMA as illustrated in Figure 5b, which selects the model out of a group of models that is the maximum a posteriori (MAP) estimate of m, e.g., (Ch. 5.3, [46]). Where BMA returns a posterior probability over the models m, BMS only returns the most probable model. In addition to the specification of the model prior of (22), BMS can be interpreted to enforce a form constraint [31] on the variable m. Specifically, we constrain the posterior distribution $q\left(m\right)$ to be a Kronecker delta function $\delta (·)$, centered around the MAP estimate of m as

$$q\left(m\right)=\delta (m-e_k),\mathrm{s}.\mathrm{t}.k=\underset{k}{arg\; max}{\overrightarrow{\mu }}_m\left(m_k=1\right){{\displaystyle \overrightarrow{{\displaystyle \mu }}}}_m\left(m_k=1\right),$$

(23)

where $e_k$ represents the k-th Euclidean standard basis vector. Figure 5b visualizes this constraint by the encircled $\delta$ on the edge corresponding to the variable m. This form constraint will effectively interrupt the flow of the messages ${\mu }_m$ and instead propagate the computed marginal distribution $q\left(m\right)$ back to the connected nodes (Theorem 3, [31]). As a result $q\left(m\right)$ will be substituted for ${\overrightarrow{\mu }}_m\left(m\right)$ in the message ${\overrightarrow{\mu }}_{s_j}\left(s_j\right)$ in Table 1, which performs a selection of the incoming messages for the outgoing message as ${\overrightarrow{\mu }}_{s_j}\left(s_j\right)={\overrightarrow{\mu }}_{s_j|m_k=1}\left(s_j\right)$.

5.3. Bayesian Model Combination

Contrary to what some consider its naming to imply, BMA does not find the best possible weighted set of models that explains the data and is therefore often subject to misinterpretation [21]. Instead, it performs a soft selection of the most probable model from the set of candidate models [7,21]. With infinite data, BMA converges to the single best model of the group of possible models [15]. In the case that the true model is inside the subset of models to evaluate, this will correctly identify the true model. However, often, the true underlying model is not within this subset and, therefore, a suboptimal model is selected. In this case, there might actually exist a specific weighted combination of models that represents the observed data better in terms of model evidence than the single best model [21].

Bayesian model combination (BMC) [7] was introduced to find the best possible weighted set of models, whilst retaining uncertainty over this weighting. The founding work of [7] presents two approaches for BMC: (1) by performing an extensive search over a discretized subspace of model weightings, and (2) by sampling from a Dirichlet distribution that extends the regular categorical model prior. Here, we illustrate the latter approach using a Dirichlet prior on $\pi$ because inference in this model can be executed efficiently using message passing.

Contrary to the previous subsection, every (set of) observation(s) is now assumed to be modeled by a distinct model $m_n$ from the set of candidate models, where n indexes the observation. Each variable $m_n$ comprises a 1-of-K binary vector with elements $m_{nk}\in \{0,1\}$ constrained by ${\sum }_{k=1}^K{m}_{nk}=1$. We specify the prior distribution

$$p\left(m_n|\pi \right)=\mathrm{Cat}\left(m_n|\pi \right),$$

(24)

where the event probabilities $\pi$ now appears as a random variable, which is modeled by

$$p\left(\pi \right)=\mathrm{Dir}\left(\pi |\alpha \right),$$

(25)

where $\alpha$ is the concentration parameters. Intuitively, the variable $\pi$ is shared among all observations, whereas $m_n$ is specific to a single observation as shown in Figure 5c.

Probabilistic Inference for Bayesian Model Combination

The exact inference in this model is intractable because the posterior over $\pi$ resembles a mixture of Dirichlet distributions with a number of components that scales exponentially with the number of observations. As a result, previous works in the literature presented approximate algorithms for performing probabilistic inference in this model, such as sampling [7]. Here, we present two alternative approaches for performing approximate inference in this model.

The first approach concerns constraining the posterior distributions over $m_n$ to be Kronecker delta functions $\delta (·)$, similar to that shown in Section 5.2 as

$$q\left(m_n\right)=\delta (m_n-e_k),\mathrm{s}.\mathrm{t}.k=\underset{k}{arg\; max}{\overrightarrow{\mu }}_{m_n}(m_{nk}=1){{\displaystyle \overrightarrow{{\displaystyle \mu }}}}_{m_n}(m_{nk}=1).$$

(26)

Here, we choose the approximate posterior $q\left(m_n\right)$ to be centered around the MAP estimate of $m_n$; however, alternative centers can also be chosen, for example, by sampling from ${\overrightarrow{\mu }}_{m_n}(m_{nk}=1){\overrightarrow{\mu }}_{m_n}(m_{nk}=1)$. Using this constraint, the backward message ${\overrightarrow{\mu }}_{\pi }\left(\pi \right)$ towards $\pi$ can be computed analytically (Appendix A.5, [47]). Batch or offline processing can be performed by an iterative message-passing procedure, similar to variational message passing [30,34,40], which requires the initialization of messages ${\overrightarrow{\mu }}_{m_n}\left(m_n\right)$ or marginals $q\left(m_n\right)$ in order to break circular dependencies between messages and marginals in the model. However, this approach also lends itself toward an online setting with streaming observations. In the online setting, however, the results are heavily influenced by prior $p\left(\pi \right)$ if chosen uninformatively as we detail in Section 6.1. In Section 6.1, we also describe an approach to cope with this initialization problem.

An alternative approach to performing approximate inference in an offline manner is obtained by variational message passing [30,34,40]. The true posterior distribution $p(\pi ,m_1,\dots ,m_N|D)$ is, in this case, approximated by the variational posterior distribution $q(\pi ,m_1,\dots ,m_N)$, being subject to naive mean-field factorization as

$$p(\pi ,m_1,\dots ,m_N|D)\approx q(\pi ,m_1,\dots ,m_N)=q\left(\pi \right)\prod _{n=1}^{N}q\left(m_n\right),$$

(27)

where the individual variational distributions are constrained to have the functional forms

$$q\left(\pi \right)=\mathrm{Dir}\left(\pi |\tilde{\alpha }\right),$$

(28a)

$$q\left(m_n\right)=\mathrm{Cat}\left(m_n|{\tilde{\pi }}_n\right),$$

(28b)

where the $\widetilde{·}$ accent is used to indicate the parameters of the variational posterior distributions. Variational message passing minimizes the variational free energy by iterating the computation of variational messages and posteriors until convergence. The corresponding variational message-passing update rules are derived in (Appendix A.5, [47]).

6. Experiments

In this section, a set of experiments is presented for the previously presented message passing-based model comparison techniques. Section 6.1 verifies the basic operations of the inference procedures for model averaging, selection and combination for data generated from a known mixture distribution. In Section 6.2, the model comparison approaches are validated on application-based examples.

All experiments were performed using the scientific programming language Julia [48] with the state-of-the-art probabilistic programming package RxInfer.jl [9]. The mixture node specified in Section 4.2 was integrated in its message-passing engine ReactiveMP.jl [49,50]. Aside from the results presented in the upcoming subsections, interactive Pluto.jl notebooks are available online (all experiments are publicly available at https://github.com/biaslab/AutomatingModelComparison, (accessed on 9 June 2023)), allowing the reader to change hyperparameters in real time.

6.1. Verification Experiments

For verification of the mixture node in Table 1, $N=\{1,5,10,100,1000\}$ observations $y_n$ are generated from the mixture distribution

$$p\left(y_n\right)=0.2\mathcal{N}(y_n|-3,1+{\sigma }^2)+0.5\mathcal{N}(y_n|0,1+{\sigma }^2)+0.3\mathcal{N}(y_n|4,1+{\sigma }^2),$$

(29)

where $\mathcal{N}(y_n|\mu ,{\sigma }^2)$ represents a normal distribution with mean $\mu$ and variance ${\sigma }^2$. ${\sigma }^2$ represents the additional observation noise variance. For the obtained data, we construct the probabilistic model

$$\begin{array}{cc}\hfill p\left(x_n|m\right)& =\mathcal{N}{(x_n|-3,1)}^{m_1}\mathcal{N}{(x_n|0,1)}^{m_2}\mathcal{N}{(x_n|4,1)}^{m_3},\hfill \end{array}$$

(30a)

$$\begin{array}{cc}\hfill p\left(y_n|x_n\right)& =\mathcal{N}(y_n|x_n,{\sigma }^2),\hfill \end{array}$$

(30b)

which is completed by the structures imposed on m as introduced in Section 5. Depending on the comparison method as outlined in Section 5.1, Section 5.2 and Section 5.3, we add an uninformative categorical prior on z or an uninformative Dirichlet prior on the event probabilities $\pi$ that model z. The aim is to infer the marginal (approximate) posterior distributions over component assignment variable z, for model averaging and selection, and over event probabilities $\pi$, for model combination.

The Bayesian model combination preliminary experiments show that the results relied significantly on the initial prior $p\left(\pi \right)$ in the online setting. Choosing this term to be uninformative, i.e., ${\alpha }_k<<1\forall k$ and ${\alpha }_i={\alpha }_j\forall i,j$, led to a posterior distribution which became dominated by the inferred cluster assignment of the first observation. As a result, the predictive class probability approached a delta distribution, centered around the class label of the first observation, leading to all consecutive observations being assigned to the same cluster. This observation is as expected, as the concentration parameters $\tilde{\alpha }$ of the posterior distribution $q\left(\pi \right)$ after the first model assignment $m_{1k}=1$ were updated as $\tilde{\alpha }=\alpha +m_1$, with prior concentration parameters $\alpha$. When the entries of $\alpha$ are small, this update will have a significant effect of the updated prior distribution over $\pi$ and consecutively over the prior belief over the model assignment ${\overrightarrow{\mu }}_{m_n}\left(m_n\right)=\mathrm{Cat}\left(m_n|\alpha /{\sum }_{k=1}^K\alpha \right)$. To remedy this undesirable behavior, the prior $p\left(\pi \right)$ was chosen to prefer uniformly distributed class labels, i.e., ${\alpha }_k>>1\forall k$ and ${\alpha }_i={\alpha }_j\forall i,j$. Although this prior yields the same forward message ${\overrightarrow{\mu }}_{m_n}\left(m_n\right)$, consecutive forward messages will be less affected by the selected models $m_n$. After the inference procedure was completed, the informativeness of this prior was removed using Bayesian model reduction [43,44], where the approximate posterior over $\pi$ was recomputed based on an alternative uninformative prior.

Figure 6 shows the inferred posterior distributions of z or the predictive distributions for z obtained from the posterior distributions $q\left(\pi \right)$, for an observation noise variance ${\sigma }^2=5$. From the results, it can be observed that Bayesian model averaging converges with increasing data to a single cluster as expected. This selected cluster corresponds to the cluster inferred by Bayesian model selection, which also corresponds to the cluster with the highest mixing weight in (29). Contrary to Bayesian model selection, the alternative event probabilities obtained with Bayesian model averaging are non-zero. Both Bayesian model combination approaches do not converge to a single cluster assignment as expected. Instead, they better recover the data-generating mixing weights in the data-generating distribution. It can be seen that the variational approach to model combination is better capable of retrieving the original mixing weights, despite the high noise variance of ${\sigma }^2=5$. The online model combination approach is less capable of retrieving the original mixing weights. This is also as expected since the online approach performs an approximate filtering procedure, contrary to the approximate smoothing procedure of the variational approach. For smaller values of the noise variance, we observe in our experiments that the online model combination strategy approaches the variational strategy.

6.2. Validation Experiments

Aside from verifying the correctness of the message-passing implementations of Section 5 using the mixture node, this section further illustrates its usefulness in a set of validation experiments, covering real-world problems.

6.2.1. Mixed Models

In order to illustrate an application of the mixture node from Table 1, we show how it can be used in a mixed model, where it connects continuous to discrete variables. Consider the hypothetical situation, where we wish to fit a mixture with fixed components but unknown mixing coefficients to some set of observations. To highlight the generality of the mixture node, the mixture components are chosen to reflect shifted product distributions, where the possible shifts are limited to a discrete set of values. The assumed probabilistic model of a single observation y is given by

$$\begin{array}{cc}\hfill p\left(a\right)& =\mathcal{N}(a|0.5,1),\hfill \end{array}$$

(31a)

$$\begin{array}{cc}\hfill p\left(b\right)& =\mathcal{N}(b|0,1),\hfill \end{array}$$

(31b)

$$\begin{array}{cc}\hfill p(c\mid z)& =\delta {(c+0.2)}^{z_1}\delta {(c+1.8)}^{z_2}\delta {(c-0.9)}^{z_3},\hfill \end{array}$$

(31c)

$$\begin{array}{cc}\hfill p\left(z\right)& =\mathrm{Cat}(z|1_3/3),\hfill \end{array}$$

(31d)

$$\begin{array}{cc}\hfill p(y|a,b,c)& =\delta (y-(ab+c\left)\right).\hfill \end{array}$$

(31e)

The variables a and b are latent variables defining the product distribution. c specifies the shift introduced on this distribution, which is picked by the selector variable z, comprising a 1-of-3 binary vector with elements $z_k\in \{0,1\}$ constrained by ${\sum }_{k=1}^3{z}_k=1$. $1_K$ denotes a vector of ones of length K. The goal is to infer the posterior distribution of z and to, therefore, fit this exotic mixture model to some set of data.

We perform offline probabilistic inference in this model using Bayesian model averaging and Bayesian model combination. For the latter approach, we extend the prior on z with a Dirichlet distribution following Section 5.3 and by assuming a variational mean-field factorization. The shifted product distributions do not yield tractable closed-form messages; therefore, these distributions are approximated following [51]. Figure 7 shows the obtained data fit on a data set of 1500 observations drawn from a standard normal distribution. This distribution does not reflect the used model in (31) on purpose to illustrate its behavior when the true underlying model is not one of the components. As expected, model averaging converges to the most dominant component, whereas model combination attempts to improve the fit by combining the different components with fixed shifts.

6.2.2. Voice Activity Detection

In this section, we illustrate a message-passing approach to voice activity detection in speech that is corrupted by additive white Gaussian noise using the mixture node from Table 1. We model speech signal $s_t$ as a first-order auto-regressive process as

$$p\left(s_t|s_{t-1}\right)=\mathcal{N}(s_t|\rho s_{t-1},{\sigma }^2),$$

(32)

with auto-regressive parameter $\rho$ and process noise variance ${\sigma }^2$. The absence of speech is modeled by independent and identically distributed variables $n_t$, which are enforced to be close to 0 as

$$p\left(n_t\right)=\mathcal{N}(n_t|0,0.01).$$

(33)

We model our observations by the mixture distribution, where we include the corruption from the additive white Gaussian noise as

$$p(y_t|s_t,n_t,z_t)=\mathcal{N}{(y_t|s_t,0.5)}^{z_{t1}}\mathcal{N}{(y_t|n_t,0.5)}^{z_{t2}}.$$

(34)

Here, $z_t$ indicates the voice activity of the observed signal as a 1-of-2 binary vector with elements $z_{tk}\in \{0,1\}$ constrained by ${\sum }_{k=1}^2{z}_{tk}=1$. Because periods of speech are often preceded by more speech, we add temporal dynamics to $z_t$ as

$$p\left(z_t|z_{t-1}\right)=\mathrm{Cat}\left(z_t|\mathrm{T}{z}_{t-1}\right),$$

(35)

where the transition matrix is specified as $\mathrm{T}=[0.99999,{10}^{-5};{10}^{-5},0.99999]$.

Figure 8 shows the clean and corrupted audio signals. The audio is sampled with a sampling frequency of 16 kHz. The corrupted signal is used for inferring $z_t$, which is presented in the bottom plot. Despite the corruption inflicted on the audio signal, this presented simple model is capable of detecting voice effectively as illustrated in the bottom plot of Figure 8.

7. Discussion

The unifying view between probabilistic inference and model comparison as presented by this paper allows us to leverage the efficient message-passing schemes for both tasks. Interestingly, this view allows for the use of belief propagation [32], variational message passing [30,34,40] and other message passing-based algorithms around the subgraph connected to the model selection variable m. This insight gives rise to a novel class of model comparison algorithms, where the prior on the model selection variable is no longer constrained to be a categorical distribution but where we now can straightforwardly introduce hierarchical and/or temporal dynamics. Furthermore, a consequence of the automatability of the message-passing algorithms is that these model comparison algorithms can easily and efficiently be implemented, without the need for error-prone and time-consuming manual derivations.

Although this paper solely focused on message passing-based probabilistic inference, we envision interesting directions for alternative probabilistic programming packages, such as Stan [14], Pyro [11], Turing [10], UltraNest [12], and PyMC [13]. Currently, only the PyMC framework allows for model comparison through their compare() function. However, often these packages allow for estimating the (log-)evidence through sampling, or for computing the evidence lower bound (ELBO), which resembles the negative VFE of (10), which is optimized using stochastic variational inference [52]. An interesting direction of future research would be to use these estimates to construct the factor node $f\left(m\right)$ in (14), with which novel model comparison algorithms can be designed, for example, where the model selection variables becomes observation dependent as in [25].

The presented approach is especially convenient when the model allows for the use of scale factors (Ch.6, [38,39]). In this way, we can efficiently compute the model evidence as shown in [39]. The introduced mixture node in Table 1 consecutively enables a simple model specification as illustrated in the Supplementary Materials source code of our experiments (all experiments are publicly available at https://github.com/biaslab/AutomatingModelComparison, (accessed on 9 June 2023)).

A limitation of the scale factors is that they can only be efficiently computed when the model submits to exact inference [39]. Extensions of the scale factors towards a variational setting would allow the use of the mixture node with a bigger variety of models. If this limitation is resolved, then the introduced approach can be combined with more complicated models, such as, for example, Bayesian neural networks, whose performance is measured by the variational free energy, see, e.g., [53,54]. This provides a novel solution to multi-task machine learning problems, where the number of tasks is not known beforehand [55]. Each Bayesian neural network can then be trained for a specific task, and additional components or networks can be added if appropriate.

The mixture nodes presented in this paper can also be nested on top of each other. As a result, hierarchical mixture models can be realized, which can quickly increase the complexity of the nested model. The question quickly arises as to where to stop. An answer to this question is provided by Bayesian model reduction [43,44]. Bayesian model reduction allows for the efficient computation of the model evidence when parts of the hierarchical model are pruned. This approach allows for the pruning of hierarchical models in an effort to bound the complexity of the entire model.

8. Conclusions

This paper bridges the gap between probabilistic inference for states and parameters, and for model comparison, allowing for the simultaneous automation of both tasks. It is shown that model comparison can be performed by message passing on a graph terminated by a node that captures the performance results of the different submodels as motivated from a variational free energy perspective. In the case where the model submits to exact inference, we can efficiently implement model comparison using our newly proposed mixture node, which leverages the efficiently computed scale factors. Based on this node description, we show how to automate Bayesian model averaging, selection, and combination by changing the (hierarchical) prior and posterior constraints on the selection variable.