Message Passing-Based Inference for Time-Varying Autoregressive Models

Abstract

Time-varying autoregressive (TVAR) models are widely used for modeling of non-stationary signals. Unfortunately, online joint adaptation of both states and parameters in these models remains a challenge. In this paper, we represent the TVAR model by a factor graph and solve the inference problem by automated message passing-based inference for states and parameters. We derive structured variational update rules for a composite “AR node” with probabilistic observations that can be used as a plug-in module in hierarchical models, for example, to model the time-varying behavior of the hyper-parameters of a time-varying AR model. Our method includes tracking of variational free energy (FE) as a Bayesian measure of TVAR model performance. The proposed methods are verified on a synthetic data set and validated on real-world data from temperature modeling and speech enhancement tasks.

1. Introduction

Autoregressive (AR) models are capable of describing a wide range of time series patterns [1,2]. The extension to Time-Varying AR (TVAR) models, where the AR coefficients are allowed to vary over time, supports tracking of non-stationary signals. TVAR models have been successfully applied to a wide range of applications, including speech signal processing [3,4,5], signature verification [6], cardiovascular response modeling [7], acoustic signature recognition of vehicles [8], radar signal processing [9], and EEG analysis [10,11].

The realization of TVAR models in practice often poses some computational issues. For many applications, such as speech processing in a hearing aid, both a low computational load and high modeling accuracy are essential.

The problem of parameter tracking in TVAR models has been extensively explored in a non-Bayesian setting. For example, ref. [12] employs over-determined modified Yule-Walker equations and [13] applies the covariance method to track the parameters in a TVAR model. In [14], expressions for the mean vector and covariance matrix of TVAR model coefficients are derived and [15] uses wavelets for TVAR model identification. Essentially, all these approaches provide maximum likelihood estimates of coefficients for TVAR models without measurement noise. In [16], autoregressive parameters were estimated from noisy observations by using a recursive least-squares adaptive filter.

We take a Bayesian approach since we are also interested in tracking Bayesian evidence (or an approximation thereof) as a model performance measure. Bayesian evidence can be used to track the optimal AR model order or more generally, to compare the performance of a TVAR model to an alternative model. To date, Bayesian parameter tracking in AR models has mostly been achieved by Monte Carlo sampling methods [17,18,19,20,21]. Sampling-based inference is highly accurate, but it is very often computationally too expensive for real-time processing on wearable devices such as hearing aids, smartwatches, etc.

In this paper, we develop a low-complexity variational message passing-based (VMP) realization for tracking of states, parameters and free energy (an upper bound on Bayesian evidence) in TVAR models. All update formulas are closed-form and the complete inference process can easily be realized.

VMP is a low-complexity distributed message passing-based realization of variational Bayesian inference on a factor graph representation of the model [22,23]. Previous work on message passing-based inference for AR models include [24], but their work describes maximum likelihood estimation and therefore does not track proper posteriors and free energy. In [25], variational inference is employed to estimate the parameters of a multivariate AR model, but their work does not take advantage of the factor graph representation.

The factor graph representation that we employ in this paper provides some distinct advantages from other works on inference in TVAR models. First, a factor graph formulation is by definition completely modular and supports re-using the derived TVAR inference equations as a plug-in module in other factor graph-based models. In particular, since we allow for measurement noise in the TVAR model specification, the proposed TVAR factor can easily be used as a latent module at any level in hierarchical dynamical models. Moreover, due to the modularity, VMP update rules can easily be mixed with different update schemes such as belief propagation and expectation [26,27] in other modules, leading to hybrid message passing schemes for efficient inference in complex models. We have implemented the TVAR model in the open source and freely available factor graph toolbox ForneyLab [28].

The rest of this paper is organized as follows. In Section 2, we specify the TVAR model as a probabilistic state space model and define the inference tasks that relate to tracking of states, parameters, and Bayesian evidence. After a short discussion on the merits of using Bayesian evidence as a model performance criterion (Section 3.1), we formulate Bayesian inference in the TVAR model as a set of sequential prediction-correction processes (Section 3.2). We will realize these processes as VMP update rules and proceed with a short review of Forney-style factor graphs and message passing in Section 4. Then, in Section 5, the VMP equations are worked out for the TVAR model and summarized in Table 1. Section 6 discusses a verification experiment on a synthetic data set and applications of the proposed TVAR model to temperature prediction and speech enhancement problems. Full derivations of the closed-form VMP update rules are presented in Appendix A.

Table 1. Variational message update rules for the autoregressive (AR) node (dashed box) of Equation (30).

VMP for the Composite AR Node
Outgoing messages	Incoming messages
$\begin{array}{cc}\hfill & \overrightarrow{\nu }\left(\mathit{y}\right)\propto \mathcal{N}\left(\mathit{y}|{\mathit{z}}_0,\Sigma \right)\hfill \\ & \overleftarrow{\nu }\left(\mathit{x}\right)\propto \mathcal{N}\left(\mathit{x}|{\mathsf{\Lambda }}_1^{-1}{\mathit{z}}_1,{\mathsf{\Lambda }}_1^{-1}\right)\hfill \\ & \overleftarrow{\nu }\left(\mathbf{\theta }\right)\propto \mathcal{N}\left(\mathbf{\theta }|{\mathsf{\Lambda }}_2^{-1}{\mathit{z}}_2,{\mathsf{\Lambda }}_2^{-1}\right)\hfill \\ & \overleftarrow{\nu }\left(\gamma \right)\propto \mathsf{\Gamma }\left(\gamma |\frac{3}{2},\frac{b}{2}\right)\hfill \end{array}$	$\begin{array}{cc}& \overleftarrow{\nu }\left(\mathit{y}\right)\propto \mathcal{N}\left(\mathit{y}|m_{\mathit{y}},V_{\mathit{y}}\right)\hfill \\ & \overrightarrow{\nu }\left(\mathit{x}\right)\propto \mathcal{N}\left(\mathit{x}|m_{\mathit{x}},V_{\mathit{x}}\right)\hfill \\ & \overrightarrow{\nu }\left(\mathbf{\theta }\right)\propto \mathcal{N}\left(\mathbf{\theta }|m_{\mathbf{\theta }},V_{\mathbf{\theta }}\right)\hfill \\ & \overrightarrow{\nu }\left(\gamma \right)\propto \mathsf{\Gamma }\left(\gamma |\alpha ,\beta \right)\hfill \end{array}$
Joint marginal $q(\mathit{y},\mathit{x})$
$q(\mathit{y},\mathit{x})\propto \overrightarrow{\nu }\left(\mathit{x}\right)exp\left[{\mathbb{E}}_{q\left(\gamma \right)q\left(\mathbf{\theta }\right)}logf(\mathit{y}\mathit{x},\mathbf{\theta },\gamma )\right]\overleftarrow{\nu }\left(\mathit{y}\right)\propto \mathcal{N}\left(\left[\begin{array}{c}\mathit{y}\\ \mathit{x}\end{array}\right]|\left[\begin{array}{c}{m}_{\mathit{y}}^{*}\\ m_{\mathit{x}}^{*}\end{array}\right],\left[\begin{array}{cc}{V}_{\mathit{y}}^{*}& V_{\mathit{yx}}^{*}\\ V_{\mathit{xy}}^{*}& V_{\mathit{x}}^{*}\end{array}\right]\right)$ (Appendix A.7)
Free energy$F\left[q\right]$
$\begin{array}{c}\hfill F\left[q\right]=u+\frac{1}{2}log2\pi +\frac{m_{\gamma }}{2}\left({\sigma }_y^2+m_y^2-2\left[V_{y{\mathit{x}}^{⊺}}+m_y{m}_{\mathit{x}}^{⊺}\right]m_{\mathbf{\theta }}+\mathrm{tr}\left[(V_{\mathbf{\theta }}+m_{\mathbf{\theta }}{m}_{\mathbf{\theta }}^{⊺})V_{\mathit{x}}^{*}\right]+m_{\mathbf{\theta }}^{⊺}(V_{\mathit{x}}^{*}+m_{\mathit{x}}^{*}{\left(m_{\mathit{x}}^{*}\right)}^{⊺})m_{\mathbf{\theta }}\right)\end{array}$
Auxiliary variables
$\begin{array}{cc}& \Sigma =m_{\mathit{A}}{(V_{\mathit{x}}^{-1}+m_{\gamma }{V}_{\mathbf{\theta }})}^{-1}{m}_{\mathit{A}}^{⊺}+m_{\mathit{V}}{\mathit{z}}_0=m_{\mathit{A}}{(V_{\mathit{x}}^{-1}+m_{\gamma }{V}_{\mathbf{\theta }})}^{-1}{V}_{\mathit{x}}^{-1}{m}_{\mathit{x}}\hfill \\ & {\mathsf{\Lambda }}_1=m_{\mathit{A}}^{⊺}{\left(V_{\mathit{y}}+m_{\mathit{V}}\right)}^{-1}{m}_{\mathit{A}}+m_{\gamma }{V}_{\theta }{\mathit{z}}_1=m_{\mathit{A}}^{⊺}{\left(V_{\mathit{y}}+m_{\mathit{V}}\right)}^{-1}{m}_{\mathit{y}}\hfill \\ & {\mathsf{\Lambda }}_2=m_{\gamma }(V_{\mathit{x}}^{*}+m_{\mathit{x}}^{*}{\left(m_{\mathit{x}}^{*}\right)}^{⊺}){\mathit{z}}_2=(V_{\mathit{xy}}^{*}+m_{\mathit{x}}^{*}{\left(m_{\mathit{y}}^{*}\right)}^{⊺})\mathit{c}{m}_{\gamma }\hfill \\ & b={\mathit{c}}^{⊺}\left[V_{\mathit{y}}^{*}+m_{\mathit{y}}^{*}{\left(m_{\mathit{y}}^{*}\right)}^{⊺}-2m_{\mathit{A}}(V_{\mathit{xy}}^{*}+m_{\mathit{x}}^{*}{\left(m_{\mathit{y}}^{*}\right)}^{⊺})+m_{\mathit{A}}(V_{\mathit{x}}^{*}+m_{\mathit{x}}^{*}{\left(m_{\mathit{x}}^{*}\right)}^{⊺})m_{\mathit{A}}^{⊺}+\mathrm{tr}\left(V_{\theta }\left(V_{\mathit{x}}^{*}+m_{\mathit{x}}^{*}{\left(m_{\mathit{x}}^{*}\right)}^{⊺}\right)\right)\right]\mathit{c}\hfill \\ & m_{\gamma }=\frac{\alpha }{\beta }{m}_{\mathit{A}}=m_{A\left(\mathbf{\theta }\right)}\hfill \\ & u=-\frac{1}{2}\left[\psi \left(\alpha \right)-log\beta \right]+\frac{1}{2}log2\pi {\sigma }_y^2={\mathit{c}}^{⊺}{V}_{\mathit{y}}^{*}\mathit{c}{m}_y={\mathit{c}}^{⊺}{m}_{\mathit{y}}^{*}\mathit{c}{V}_{y\mathit{x}}=V_{\mathit{yx}}^{*}\mathit{c}\hfill \end{array}$

2. Model Specification and Problem Definition

In this section, we first specify TVAR as a state-space model. This is followed by an inference problem formulation.

2.1. Model Specification

A TVAR model is specified as

$$\begin{array}{cc}\hfill & {\theta }_{m,t}\sim \mathcal{N}({\theta }_{m,t-1},\omega )\hfill \end{array}$$

(1a)

$$\begin{array}{cc}\hfill & x_t\sim \mathcal{N}\left(\sum _{m=1}^M{\theta }_{m,t}{x}_{t-m},{\gamma }^{-1}\right)\hfill \end{array}$$

(1b)

$$\begin{array}{cc}\hfill & y_t\sim \mathcal{N}(x_t,\tau ),\hfill \end{array}$$

(1c)

where $y_t\in \mathbb{R}$, $x_t\in \mathbb{R}$ and ${\theta }_{m,t}\in \mathbb{R}$ represent the the observation, state and parameters at time t, respectively. M denotes the order of the AR model. As a notational convention, we use $\mathcal{N}(\mu ,\Sigma )$ to denote a Gaussian distribution with mean $\mu$ and co-variance matrix $\Sigma$. We can re-write (1) in state-space form as

$$\begin{array}{cc}\hfill {\mathbf{\theta }}_t& \sim \mathcal{N}({\mathbf{\theta }}_{t-1},\omega {\mathrm{I}}_M)\hfill \end{array}$$

(2a)

$$\begin{array}{cc}\hfill {\mathit{x}}_t& \sim \mathcal{N}\left(A\left({\mathbf{\theta }}_t\right){\mathit{x}}_{t-1},V\left(\gamma \right)\right)\hfill \end{array}$$

(2b)

$$\begin{array}{cc}\hfill y_t& \sim \mathcal{N}({\mathit{c}}^{⊺}{\mathit{x}}_t,\tau ),\hfill \end{array}$$

(2c)

where ${\mathbf{\theta }}_t={({\theta }_{m,t},{\theta }_{m-1,t},\cdots ,{\theta }_{m-M,t})}^{⊺}$, ${\mathit{x}}_{\mathbf{t}}={(x_t,x_{t-1},\cdots ,x_{t-M+1})}^{⊺}$, $\mathit{c}={(1,0,\dots ,0)}^{⊺}$ is an M-dimensional unit vector, $V\left(\gamma \right)=(1/\gamma )\mathit{c}{\mathit{c}}^{⊺}$, and

$$A\left(\mathbf{\theta }\right)=\left[\begin{array}{c}{\mathbf{\theta }}^{⊺}\\ {\mathrm{I}}_{M-1}& \mathbf{0}.\end{array}\right]$$

(3)

Technically, a TVAR model usually assumes $\tau =0$ indicating that there is no measurement noise. Note that the presence of measurement noise in (2c) “hides” the states ${\mathit{x}}_t$ in the generative model (2) from the observation sequence $y_t$, yielding a latent TVAR. We add measurement noise explicitly, so the model is able to accept information from likelihood functions that are not constrained to be delta functions with hard observations. As a result, the AR model that we define here can be used at any level in deep hierarchical structures such as [29] as a plug-in module.

In a time-invariant AR model, $\mathbf{\theta }$ are part of the parameters of the system. In a time-varying AR model, we consider ${\mathbf{\theta }}_t$ and ${\mathit{x}}_t$ together the set of time-varying states. The parameters of the TVAR model are $\{{\mathbf{\theta }}_0,{\mathit{x}}_0,\omega ,\gamma ,\tau \}$.

At the heart of the TVAR model is the transition model (2b), where $A\left({\mathbf{\theta }}_t\right)$ is a companion matrix with AR coefficients. The multiplication $A\left(\mathbf{\theta }\right){\mathit{x}}_{t-1}$ performs two operations: a dot product ${\mathbf{\theta }}_t^{⊺}{\mathit{x}}_{t-1}$ and a vector shift of ${\mathit{x}}_{t-1}$ by one time step. The latter operation can be interpreted as bookkeeping, as it shifts each entry of ${\mathit{x}}_{t-1}$ one position down and discards $x_{t-M}$.

2.2. Problem Definition

For a given time series $\mathit{y}=(y_1,y_2,\dots ,y_T)$, we are firstly interested in recursively updating posteriors for the states $p\left({\mathit{x}}_t\right|{\mathit{y}}_{1:l})$ and $p\left({\mathbf{\theta }}_t\right|{\mathit{y}}_{1:l})$. In this context, prediction, filtering and smoothing are recovered for $l<t$, $l=t$ and $l>t$, respectively.

Furthermore, we are interested in computing posteriors for the parameters $p\left({\mathbf{\theta }}_0\right|\mathit{y})$, $p\left({\mathit{x}}_0\right|\mathit{y})$, $p\left(\omega \right|\mathit{y})$, $p\left(\gamma \right|\mathit{y})$ and $p\left(\tau \right|\mathit{y})$.

Finally, we are interested in scoring the performance of a proposed TVAR model m with specified priors for the parameters. In this paper, we take a full Bayesian approach and select Bayesian evidence $p\left(\mathit{y}\right|m)$ as the performance criterion. Section 3.1 discusses the merits of Bayesian evidence as a model performance criterion.

3. Inference in TVAR Models

In this section, we first discuss some of the merits of using Bayesian evidence as a model performance criterion. This is followed by an exposition of how to compute Bayesian evidence and the desired posteriors in the TVAR model.

3.1. Bayesian Evidence as a Model Performance Criterion

Consider a model m with parameters $\theta$ and observations y. Bayesian evidence $p\left(\mathit{y}\right|m)$ is considered an excellent model performance criterion. Note the following decomposition [30]:

$$\begin{array}{cc}\hfill logp\left(\mathit{y}\right|m)& =log\frac{p\left(\mathit{y}\right|\mathbf{\theta },m\left)p\right(\mathbf{\theta }\left|m\right)}{p\left(\mathbf{\theta }\right|\mathit{y},m)}(\mathrm{use}\mathrm{Bayes}\mathrm{rule})\hfill \\ \hfill & =\int p\left(\mathbf{\theta }\right|\mathit{y},m)·\underset{logp\left(\mathit{y}\right|m)\mathrm{is}\mathrm{not}\mathrm{a}\mathrm{function}\mathrm{of}\mathbf{\theta }}{\underbrace{log\frac{p\left(\mathit{y}\right|\mathbf{\theta },m\left)p\right(\mathbf{\theta }\left|m\right)}{p\left(\mathbf{\theta }\right|\mathit{y},m)}}}\mathrm{d}\mathbf{\theta }\hfill \\ \hfill & =\underset{\mathrm{data}\mathrm{fit}}{\underbrace{\int p\left(\mathbf{\theta }\right|\mathit{y},m)logp(\mathit{y}|\mathbf{\theta },m)\mathrm{d}\mathbf{\theta }}}-\underset{\mathrm{complexity}}{\underbrace{\int p\left(\mathbf{\theta }\right|\mathit{y},m)log\frac{p\left(\mathbf{\theta }\right|\mathit{y},m)}{p\left(\mathbf{\theta }\right|m)}\mathrm{d}\mathbf{\theta }}}\hfill \end{array}$$

(4)

The first term (data fit or sometimes called accuracy) measures how well the model predicts the data $\mathit{y}$, after having learned from the data. We want this term to be large (although only focusing on this term could lead to over-fitting). The second term (complexity) quantifies the amount of information that the model absorbed through learning by moving parameter beliefs from $p\left(\mathbf{\theta }\right|m)$ to $p\left(\mathbf{\theta }\right|\mathit{y},m)$. To see this, note that the mutual information between two variables $\mathbf{\theta }$ and $\mathit{y}$, which is defined as

$$I[\mathbf{\theta };\mathit{y}]=\int \int p(\mathbf{\theta },\mathit{y})log\frac{p\left(\mathbf{\theta }\right|\mathit{y})}{p\left(\mathbf{\theta }\right)}\mathrm{d}\mathbf{\theta }\mathrm{d}\mathit{y},$$

can be interpreted as expected complexity. The complexity term regularizes the Bayesian learning process automatically. Preference for models with high Bayesian evidence implies a preference for models that get a good data fit without the need to learn much from the data set. These types of models are said to generalize well, since they can be applied to different data sets without specific adaptations for each data set. Therefore, Bayesian learning automatically leads to models that tend to generalize well.

Note that Bayesian evidence for a model m, given a full times series $\mathit{y}=(y_1,y_2,\dots ,y_T)$, can be computed by multiplication of the sample-based evidences:

$$p\left(\mathit{y}\right|m)=\prod _{t=1}^{T}p\left(y_t\right|{\mathit{y}}_{1:t-1},m).$$

(5)

3.2. Inference as a Prediction-Correction Process

To illustrate the type of calculations that are needed for computing Bayesian model evidence and the posteriors for states and parameters, we now proceed to write out the needed calculations for the TVAR model in a filtering context.

Assume that at the beginning of time step t, we are given the state posteriors $q\left({\mathit{x}}_{t-1}\right|{\mathit{y}}_{1:t-1})$, $q\left({\mathbf{\theta }}_{t-1}\right|{\mathit{y}}_{1:t-1})$. We will denote the inferred probabilities by $q(·)$, in contrast to factors from the generative model that are written as $p(·)$. We start the procedure by setting the state priors for the generative model at step t to the posteriors of the previous time step

$$\begin{array}{cc}\hfill p\left({\mathit{x}}_{t-1}\right|{\mathit{y}}_{1:t-1})& :=q\left({\mathit{x}}_{t-1}\right|{\mathit{y}}_{1:t-1})\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill p\left({\mathbf{\theta }}_{t-1}\right|{\mathit{y}}_{1:t-1})& :=q\left({\mathbf{\theta }}_{t-1}\right|{\mathit{y}}_{1:t-1})\hfill \end{array}$$

(7)

Given a new observation $y_t$, we are now interested inferring the evidence $q\left(y_t\right|{\mathit{y}}_{t-1})$, and in inferring posteriors $q\left({\mathit{x}}_t\right|{\mathit{y}}_{1:t})$ and $q\left({\mathbf{\theta }}_t\right|{\mathit{y}}_{1:t})$.

This involves a prediction (forward) pass through the system that leads to the evidence update, followed by a correction (backward) pass that updates the states. We work this out in detail below. For clarity of exposition, in this section we call ${\mathit{x}}_t$ states and ${\mathbf{\theta }}_t$ parameters. Starting with the forward pass (from latent variables toward observation), we first compute a parameter prior predictive:

$$\underset{\begin{array}{c}\mathrm{parameter}\\ \mathrm{prior}\mathrm{predictive}\end{array}}{\underbrace{q\left({\mathbf{\theta }}_t\right|{\mathit{y}}_{1:t-1})}}=\int \underset{\begin{array}{c}\mathrm{parameter}\\ \mathrm{transition}\end{array}}{\underbrace{p\left({\mathbf{\theta }}_t\right|{\mathbf{\theta }}_{t-1})}}\underset{\begin{array}{c}\mathrm{parameter}\\ \mathrm{prior}\end{array}}{\underbrace{p\left({\mathbf{\theta }}_{t-1}\right|{\mathit{y}}_{1:t-1})}}\mathrm{d}{\mathbf{\theta }}_{t-1}.$$

(8)

Then the prior predictive for the state transition becomes:

$$\underset{\begin{array}{c}\mathrm{state}\mathrm{transition}\\ \mathrm{prior}\mathrm{predictive}\end{array}}{\underbrace{q\left({\mathit{x}}_t\right|{\mathit{x}}_{t-1},{\mathit{y}}_{1:t-1})}}=\int \underset{\mathrm{state}\mathrm{transition}}{\underbrace{p\left({\mathit{x}}_t\right|{\mathit{x}}_{t-1},{\mathbf{\theta }}_t)}}\underset{\begin{array}{c}\mathrm{parameter}\\ \mathrm{prior}\mathrm{predictive}\end{array}}{\underbrace{q\left({\mathbf{\theta }}_t\right|{\mathit{y}}_{1:t-1})}}\mathrm{d}{\mathbf{\theta }}_t.$$

(9)

Note that the state transition prior predictive, due to its dependency on time-varying ${\theta }_t$, is a function of the observed data sequence. The state transition prior predictive can be used together with the state prior to inferring the state prior predictive:

$$\underset{\begin{array}{c}\mathrm{state}\\ \mathrm{prior}\mathrm{predictive}\end{array}}{\underbrace{q\left({\mathit{x}}_t\right|{\mathit{y}}_{1:t-1})}}=\int \underset{\begin{array}{c}\mathrm{state}\mathrm{transition}\\ \mathrm{prior}\mathrm{predictive}\end{array}}{\underbrace{q\left({\mathit{x}}_t\right|{\mathit{x}}_{t-1},{\mathit{y}}_{1:t-1})}}\underset{\mathrm{state}\mathrm{prior}}{\underbrace{p\left({\mathit{x}}_{t-1}\right|{\mathit{y}}_{1:t-1})}}\mathrm{d}{\mathit{x}}_{t-1}.$$

(10)

The evidence for model m that is provided by observation $y_t$ is then given by

$$\underset{\mathrm{evidence}}{\underbrace{q\left(y_t\right|{\mathit{y}}_{1:t-1})}}=\int \underset{\begin{array}{c}\mathrm{state}\\ \mathrm{likelihood}\end{array}}{\underbrace{p\left(y_t\right|{\mathit{x}}_t)}}\underset{\begin{array}{c}\mathrm{state}\mathrm{prior}\\ \mathrm{predictive}\end{array}}{\underbrace{q\left({\mathit{x}}_t\right|{\mathit{y}}_{1:t-1})}}\mathrm{d}{\mathit{x}}_t.$$

(11)

When $y_t$ has not yet been observed, $q\left(y_t\right|{\mathit{y}}_{1:t-1})$ is a prediction for $y_t$. After plugging in the observed value for $y_t$, the evidence is a scalar that scores how well the model performed in predicting $y_t$. As discussed in (5), the results $q\left(y_t\right|{\mathit{y}}_{1:t-1})$ for $t=1,2,\dots ,T$ in (11) can be used to score the model performance for a given time series $\mathit{y}=(y_1,y_2,\dots ,y_T)$. Note that to update the evidence, we need to integrate over all latent variables ${\mathbf{\theta }}_{t-1}$, ${\mathbf{\theta }}_t$, ${\mathit{x}}_{t-1}$ and ${\mathit{x}}_t$ (by (8)–(11)). In principle, this scheme needs to be extended with integration over the parameters $\omega$, $\gamma$ and $\tau$.

Once we have inferred the evidence, we proceed by a backward corrective pass through the model to update the posterior over the latent variables given the new observation $y_t$. The state posterior can be updated by Bayes rule:

$$\underset{\mathrm{state}\mathrm{posterior}}{\underbrace{q\left({\mathit{x}}_t\right|{\mathit{y}}_{1:t})}}=\frac{\stackrel{\begin{array}{c}\mathrm{state}\\ \mathrm{likelihood}\end{array}}{\overbrace{p\left(y_t\right|{\mathit{x}}_t)}}\stackrel{\begin{array}{c}\mathrm{state}\mathrm{prior}\\ \mathrm{predictive}\end{array}}{\overbrace{q\left({\mathit{x}}_t\right|{\mathit{y}}_{1:t-1})}}}{\underset{\mathrm{evidence}}{\underbrace{q\left(y_t\right|{\mathit{y}}_{1:t-1})}}}$$

(12)

Next, we need to compute a likelihood function for the parameters. Fortunately, we can re-use some intermediate results from the forward pass. The likelihood for the parameters is given by

$$\underset{\begin{array}{c}\mathrm{parameter}\\ \mathrm{likelihood}\end{array}}{\underbrace{q\left(y_t\right|{\mathbf{\theta }}_t,{\mathit{y}}_{1:t-1})}}=\int \underset{\begin{array}{c}\mathrm{state}\\ \mathrm{likelihood}\end{array}}{\underbrace{p\left(y_t\right|{\mathit{x}}_t)}}\underset{\begin{array}{c}\mathrm{state}\mathrm{prior}\\ \mathrm{predictive}\end{array}}{\underbrace{q\left({\mathit{x}}_t\right|{\mathbf{\theta }}_t,{\mathit{y}}_{1:t-1})}}\mathrm{d}{\mathit{x}}_t$$

(13)

The parameter posterior then follows from Bayes rule:

$$\underset{\mathrm{parameter}\mathrm{posterior}}{\underbrace{q\left({\mathbf{\theta }}_t\right|{\mathit{y}}_{1:t})}}=\frac{\stackrel{\begin{array}{c}\mathrm{parameter}\\ \mathrm{likelihood}\end{array}}{\overbrace{q\left(y_t\right|{\mathbf{\theta }}_t,{\mathit{y}}_{1:t-1})}}\stackrel{\begin{array}{c}\mathrm{parameter}\\ \mathrm{prior}\mathrm{predictive}\end{array}}{\overbrace{q\left({\mathbf{\theta }}_t\right|{\mathit{y}}_{1:t-1})}}}{\underset{\mathrm{evidence}}{\underbrace{q\left(y_t\right|{\mathit{y}}_{1:t-1})}}}$$

(14)

Equations (11), (12) and (14) contain the solutions to our inference task. Note that the evidence $q\left(y_t\right|{\mathit{y}}_{1:t-1})$ is needed to normalize the latent variable posteriors in (12) and (14). Moreover, while we integrate over the states by (11) to compute the evidence, (14) reveals that the evidence can alternatively be computed by integrating over the parameters through

$$\underset{\mathrm{evidence}}{\underbrace{q\left(y_t\right|{\mathit{y}}_{1:t-1})}}=\int \underset{\begin{array}{c}\mathrm{parameter}\\ \mathrm{likelihood}\end{array}}{\underbrace{q\left(y_t\right|{\mathbf{\theta }}_t,{\mathit{y}}_{1:t-1})}}\underset{\begin{array}{c}\mathrm{parameter}\\ \mathrm{prior}\mathrm{predictive}\end{array}}{\underbrace{q\left({\mathbf{\theta }}_t\right|{\mathit{y}}_{1:t-1})}}\mathrm{d}{\mathbf{\theta }}_t.$$

(15)

This latter method of evidence computation may be useful if re-using (11) in (14) leads to numerical rounding issues.

Unfortunately, many of Equations (8) through (14) are not analytically tractable for the TVAR model. This happens due to (1) integration over large state spaces, (2) non-conjugate prior-posterior pairing, and (3) the absence of a closed-form solution for the evidence factor.

To overcome this challenge, we will perform inference by a hybrid message passing scheme in a factor graph. In the next section, we give a short review of Forney-Style Factor Graphs (FFG) and Message-Passing (MP) based inference techniques.

4. Factor Graphs and Message Passing-Based Inference

In this section, we make a brief introduction of Forney-Style Factor graph (FFG) and sum-product (SP) algorithm. After that we review the minimization of variational free energy and Variational Message Passing (VMP) algorithm.

4.1. Forney-Style Factor Graphs

A Forney-style Factor graph is a representation of a factorized function where the factors and variables are represented by nodes and edges, respectively. An edge is connected to a node if and only if the (edge) variable is an argument of the node function. In our work, we use FFGs to represent factorized probability distributions. FFGs provide both an attractive visualization of the model and a highly efficient and modular inference method based on message passing. An important component of the FFG representation is the equality node. If a variable x is shared between more than two nodes, then we introduce two auxiliary variables $x^{\prime }$ and $x^{″}$ and use an equality node

$$\begin{array}{c}\hfill f_{=}(x,x^{\prime },x^{″})=\delta (x-x^{\prime })\delta (x-x^{″})\end{array}$$

(16)

to constrain the marginal beliefs over x, $x^{\prime }$, $x^{″}$ to be equal. With this mechanism, any factorized function can be represented as an FFG.

An FFG visualization of the TVAR model is depicted in Figure 3, but for illustrative purposes, we first consider an example factorized distribution

$$\begin{array}{cc}\hfill p(x_1,x_2,x_3,x_4)& =p\left(x_1\right)p\left(x_2\right|x_1)p\left(x_3\right|x_2)p\left(x_4\right|x_3)\hfill \end{array}$$

(17)

This distribution can be visualized by an FFG shown in Figure 1. An FFG is in principle an undirected graph but we often draw arrows on the edges in the “generative” direction, which is the direction that describes how the observed data is generated. Assume that we are interested in computing the marginal for $x_2$, given by

$$\begin{array}{cc}\hfill p\left(x_2\right)& =\int \int \int p(x_1,x_2,x_3,x_4)\mathrm{d}{x}_1\mathrm{d}{x}_3\mathrm{d}{x}_4\hfill \end{array}$$

(18)

Figure 1. An FFG corresponding to model (17), including messages as per (19). For graphical clarity, we defined $f_a\left(x_1\right)=p\left(x_1\right)$, $f_b(x_1,x_2)=p\left(x_2\right|x_1)$, $f_c(x_2,x_3)=p\left(x_3\right|x_2)$ and $f_d(x_3,x_4)=p\left(x_4\right|x_3)$.

We can reduce the complexity of computing this integral by rearranging the factors over the integration signs as

$$\begin{array}{cc}\hfill p\left(x_2\right)& =\underset{{\overrightarrow{\mu }}_2\left(x_2\right)}{\underbrace{\int \underset{{\overrightarrow{\mu }}_1\left(x_1\right)}{\underbrace{p\left(x_1\right)}}p\left(x_2\right|x_1)\mathrm{d}{x}_1}}·\underset{{\overleftarrow{\mu }}_2\left(x_2\right)}{\underbrace{\left(\int p\left(x_3\right|x_2)\underset{{\overleftarrow{\mu }}_3\left(x_3\right)}{\underbrace{\left(\int p\left(x_4\right|x_3)\mathrm{d}{x}_3)\right)}}\mathrm{d}{x}_3\right)}}\hfill \end{array}$$

(19a)

$$\begin{array}{cc}\hfill & ={\overrightarrow{\mu }}_2\left(x_2\right)·{\overleftarrow{\mu }}_2\left(x_2\right).\hfill \end{array}$$

(19b)

These re-arranged integrals can be interpreted as messages that are passed over the edges, see Figure 1. It is a notational convention to call a message $\overrightarrow{\mu }(·)$ that aligns with the direction of the edge arrow a forward message and similarly, a message $\overleftarrow{\mu }(·)$ that opposes the direction of the edge is called a backward message.

This message passed-based algorithm of computing the marginal is called belief propagation (BP) or the sum-product algorithm. As can be verified in (19), for a node with factor $f(y,x_1,\cdots ,x_n)$, the outgoing BP message $\overrightarrow{\mu }\left(y\right)$ to variable y can be expressed as

$${\overrightarrow{\mu }}_y\left(y\right)=\int \cdots \int f(y,x_1,\cdots ,x_n)\prod _{i=1}^n{\overrightarrow{\mu }}_i\left(x_i\right)\mathrm{d}{x}_i.$$

(20)

where ${\overrightarrow{\mu }}_i\left(x_i\right)$ is an incoming message over edge $x_i$. If the factor graph is a tree, meaning that the graph contains no cycles, then BP leads to exact Bayesian inference. A more detailed explanation of belief propagation message passing in FFGs can be found in [26].

4.2. Free Energy and Variational Message Passing

Technically, BP is a message passing algorithm that belongs to a family of message passing algorithms that minimize a constrained variational free energy functional [31]. Unfortunately, the sum-product rule (20) only has a closed-form solution for Gaussian incoming messages ${\overrightarrow{\mu }}_i\left(x_i\right)$ and linear variable relations in $f(y,x_1,\dots ,x_n)$. Another important member of the free energy minimizing algorithms is the Variational Message Passing (VMP) algorithm [22]. VMP enjoys a wider range of analytically computable message update rules.

We shortly review variational Bayesian inference and VMP next. Consider a model $p(\mathit{y},\mathit{x})$ with observations $\mathit{y}$ and unobserved (latent) variables $\mathit{x}$. We are interested in inferring the posterior distribution $p\left(\mathit{x}\right|\mathit{y})$. In variational inference we introduce an approximate posterior $q\left(\mathit{x}\right)$ and define a variational free energy functional as

$$\begin{array}{c}\hfill F\left[q\right]\triangleq \int q\left(\mathit{x}\right)log\frac{q\left(\mathit{x}\right)}{p(\mathit{y},\mathit{x})}\mathrm{d}\mathit{x}=\underset{\mathrm{KL}\mathrm{divergence}{D}_{\mathrm{KL}}(q,p)}{\underbrace{\int q\left(\mathit{x}\right)log\frac{q\left(\mathit{x}\right)}{p\left(\mathit{x}\right|\mathit{y})}\mathrm{d}\mathit{x}}}-\underset{\log -\mathrm{evidence}}{\underbrace{logp\left(\mathit{y}\right)}}.\end{array}$$

(21)

The second term in (21) (log-evidence) is not a function of the argument of F. The first term is a KL-divergence, which is by definition non-negative and only equals zero for $q\left(\mathit{x}\right)=p\left(\mathit{x}\right|\mathit{y})$. As a result, variational inference by minimization of $F\left[q\right]$ provides

$$q^{*}\left(\mathit{x}\right)=arg\underset{q}{min}F\left[q\right]$$

(22)

which is an approximation to the Bayesian posterior $p\left(\mathit{x}\right|\mathit{y})$. Moreover, the minimized free energy $F\left[q^{*}\right]$ is an upper bound for minus log-evidence and in practice is used a model performance criterion. Similarly to (4), the free energy can be decomposed as

$$F\left[q\right]=\underset{\mathrm{accuracy}}{\underbrace{\int q\left(\mathit{x}\right)logp\left(\mathit{y}\right|\mathit{x},m)\mathrm{d}\mathit{x}}}-\underset{\mathrm{complexity}}{\underbrace{\int q\left(\mathit{x}\right)log\frac{q\left(\mathit{x}\right)}{\underset{\mathrm{prior}}{\underbrace{p\left(\mathit{x}\right|m)}}}\mathrm{d}\mathit{x}}}$$

(23)

which underwrites its usage as a performance criterion for model m, given observations $\mathit{y}$.

In an FFG context, the model $p(\mathit{y},\mathit{x})$ is represented by a set of connected nodes. Consider a generic node of the FFG, given by $f(y,x_1,\cdots ,x_n)$ where in the case of VMP, the incoming messages are approximations to the marginals $q_i\left(x_i\right),i=1,\cdots ,n$, see Figure 2.

Figure 2. A generic node $f(y,x_1,\cdots ,x_n)$ with incoming variational messages $q_i\left(x_i\right)$ and outgoing variational message $\overrightarrow{\nu }\left(y\right)$, see Equation (24). Note that the marginals $q(·)$ propagate in the graph as messages.

It can be shown that the outgoing VMP message of f towards edge y is given by [32]

$$\begin{array}{c}\hfill \overrightarrow{\nu }\left(y\right)\propto exp\left(\int ...\int logf(y,x_1,\cdots ,x_n)\prod _{i=1}q\left(x_i\right)\mathrm{d}{x}_i\right).\end{array}$$

(24)

In this paper, we adopt the notational convention to denote belief propagation messages (computed by (20)) by $\mu$ and VMP messages (computed by (24)) by $\nu$. The approximate marginal $q\left(y\right)$ can be obtained by multiplying incoming and outgoing messages on the edge for y

$$\begin{array}{c}\hfill q\left(y\right)\propto \overrightarrow{\nu }\left(y\right)\overleftarrow{\nu }\left(y\right).\end{array}$$

(25)

This process (compute forward and backward messages for an edge and update the marginal) is executed sequentially and repeatedly for all edges in the graph until convergence. In contrast to BP-based inference, the VMP and marginal update rules (24) and (25) lead to closed-form expressions for a large set of conjugate node pairs from the exponential family of distributions. For instance, updating the variance parameter of a Gaussian node with a connected inverse-gamma distribution node results in closed-form VMP updates.

In short, both BP- and VMP-based message passing can be interpreted as minimizing variational free energy, albeit under a different set of local constraints [31]. Typical constraints include factorization and form constraints on the posterior such as $q\left(\mathit{x}\right)={\prod }_i{q}_i\left(x_i\right)$ and $q\left(\mathit{x}\right)=\mathcal{N}\left(\mathit{x}\right|\mathbf{\mu },\Sigma )$, respectively. Since the constraints are local, BP and VMP can be combined in a factor graph to create hybrid message passing-based variational inference algorithms. For a more detailed explanation of VMP in FFGs, we refer to [32]. Note that hybrid message passing does in general not guarantee to minimize variational free energy [33]. However, in our experiments in Section 6 we will show that iterating our stationary solutions by message passing does lead to free energy minimization.

5. Variational Message Passing for TVAR Models

In this section, we focus on deriving message passing-based inference in the TVAR model. We develop a TVAR composite factor for the FFG framework and specify the intractable BP messages around the TVAR node. Then we present a message passing-based inference solution.

5.1. Message Passing-Based Inference in the TVAR Model

The TVAR model at time step t can be represented by an FFG as shown in Figure 3. We are interested in providing a message passing solution to the inference tasks as specified by Equations (8)–(14). At the left-hand side of Figure 3, the incoming messages are the priors $p\left({\mathbf{\theta }}_{t-1}\right|{\mathit{y}}_{1:t-1})$ and $p\left({\mathit{x}}_{t-1}\right|{\mathit{y}}_{1:t-1})$. At the bottom of the graph, there is a new observation $y_t$. The goal is to pass messages in the graph to compute posteriors $q\left({\mathbf{\theta }}_t\right|{\mathit{y}}_{1:t})$ (message ⑯) and $q\left({\mathit{x}}_t\right|{\mathit{y}}_{1:t})$ (message ⑪). In order to support smoothing algorithms, we also want to be able to pass incoming prior messages from the right-hand side to outgoing messages ⓭ and ⓲ at the left-hand side. Forward and backward messages are drawn as open and closed circles respectively.

Figure 3. One time segment of an FFG corresponding to the TVAR model. We use small black nodes to denote observations and fixed given parameter values. The observation node for $y_t$ sends a message $\delta (y_t-{\widehat{y}}_t)$ into the graph to indicate that $y_t={\widehat{y}}_t$ has been observed. Dashed undirected edges denote time-invariant variables. Circled numbers indicate a selected computation schedule. Backward messages are marked by black circles. The intractable messages are labeled with red. The dashed box represents a composite AR node as specified by (30).

Technically, the generative model (2) at time step t for the TVAR model can shortly be written as $p\left(y_t\right|{\mathit{z}}_t)p\left({\mathit{z}}_t\right|{\mathit{z}}_{t-1})$, where ${\mathit{z}}_t=\{{\mathit{x}}_t,{\mathbf{\theta }}_t,\omega ,\gamma ,\tau \}$ are the latent variables. On this view, we can write the free energy functional for the TVAR model at time step t as

$$\begin{array}{c}\hfill F\left[q({\mathit{z}}_{t-1},{\mathit{z}}_t|{\mathit{y}}_{1:t})\right]=\int \int q({\mathit{z}}_{t-1},{\mathit{z}}_t|{\mathit{y}}_{1:t})log\frac{\stackrel{\mathrm{posterior}}{\overbrace{q({\mathit{z}}_{t-1},{\mathit{z}}_t|{\mathit{y}}_{1:t})}}}{\underset{\mathrm{generative}\mathrm{model}}{\underbrace{p\left(y_t\right|{\mathit{z}}_t)p\left({\mathit{z}}_t\right|{\mathit{z}}_{t-1})}}\underset{\mathrm{prior}\mathrm{from}\mathrm{past}}{\underbrace{p\left({\mathit{z}}_{t-1}\right|{\mathit{y}}_{1:t-1})}}}\mathrm{d}{\mathit{z}}_{t-1}\mathrm{d}{\mathit{z}}_t.\end{array}$$

(26)

and minimize $F\left[q\right]$ by message passing. In a smoothing context, we would include a prior from the future $p\left({\mathit{z}}_t\right|{\mathit{y}}_{t+1:t+T}):=q\left({\mathit{z}}_t\right|{\mathit{y}}_{t+1:t+T})$, yielding a free energy functional

$$F\left[q({\mathit{z}}_{t-1},{\mathit{z}}_t|{\mathit{y}}_{1:T})\right]=\int \int q({\mathit{z}}_{t-1},{\mathit{z}}_t|{\mathit{y}}_{1:T})log\frac{\stackrel{\mathrm{posterior}}{\overbrace{q({\mathit{z}}_{t-1},{\mathit{z}}_t|{\mathit{y}}_{1:T})}}}{\underset{\mathrm{generative}\mathrm{model}}{\underbrace{p\left(y_t\right|{\mathit{z}}_t)p\left({\mathit{z}}_t\right|{\mathit{z}}_{t-1})}}\underset{\begin{array}{c}\mathrm{prior}\\ \mathrm{from}\mathrm{past}\end{array}}{\underbrace{p\left({\mathit{z}}_{t-1}\right|{\mathit{y}}_{1:t-1})}}\underset{\begin{array}{c}\mathrm{prior}\\ \mathrm{from}\mathrm{future}\end{array}}{\underbrace{p\left({\mathit{z}}_t\right|{\mathit{y}}_{t+1:t+T})}}}\mathrm{d}{\mathit{z}}_{t-1}\mathrm{d}{\mathit{z}}_t.$$

(27)

In a filtering context, $q\left({\mathit{z}}_t\right|{\mathit{y}}_{t+1:t+T})\propto 1$ and the functional (27) simplifies to (26).

5.2. Intractable Messages and the Composite AR Node

The modularity of message passing in FFGs allows us to focus on only the intractable message and marginal updates. For instance, while there is no problem with the analytical computation of the backward message ⓬, the corresponding forward message ➃,

$$\overrightarrow{\mu }\left({\mathit{x}}_t\right)=\int \mathcal{N}\left({\mathit{x}}_t|A\left({\mathbf{\theta }}_t\right){\mathit{x}}_{t-1},V\left(\gamma \right)\right)\underset{\mathrm{Gaussian}\mathrm{messages}}{\underbrace{\overrightarrow{\mu }\left({\mathit{x}}_{t-1}\right)\overrightarrow{\mu }\left({\mathbf{\theta }}_t\right)\overrightarrow{\mu }\left(\gamma \right)}}\mathrm{d}\gamma \mathrm{d}{\mathbf{\theta }}_t{\mathit{x}}_{t-1}$$

(28)

cannot be analytically solved [34]. Similarly, some other messages ⓭, ⓮ and ⓯ do not have a closed-form solution in the constrained free energy minimization framework. For purpose of identification, in Figure 3 intractable messages are marked in red color.

In an FFG framework, we can isolate the problematic part of the TVAR model (Figure 3) by introducing a “composite” AR node. Composite nodes conceal their internal operations from the rest of the graph. As a result, inference can proceed as long as each composite node follows proper message passing communication rules at its interfaces to the rest of the graph. The composite AR node

$$f_{\mathrm{AR}}({\mathit{x}}_t,{\mathit{x}}_{t-1},{\mathbf{\theta }}_t,\gamma )=\mathcal{N}\left({\mathit{x}}_t|A\left({\mathbf{\theta }}_t\right){\mathit{x}}_{t-1},V\left(\gamma \right)\right)$$

(29)

is indicated in Figure 3 by a dashed box. Note that the internal shuffling of the parameters ${\mathbf{\theta }}_t$ and $\gamma$, respectively by means of $A\left({\mathbf{\theta }}_t\right)$ and $V\left(\gamma \right)$, is hidden from the network outside the composite AR node.

5.3. VMP Update Rules for the Composite AR Node

We isolate the composite AR node by the specification

$$f_{\mathrm{AR}}(\mathit{y},\mathit{x},\mathbf{\theta },\gamma )=\mathcal{N}\left(\mathit{y}\left|A\right(\mathbf{\theta })\mathit{x},V(\gamma )\right),$$

(30)

where, relative to (29), we used substitutions $\mathit{y}={\mathit{x}}_t,\mathit{x}={\mathit{x}}_{t-1},\mathbf{\theta }={\mathbf{\theta }}_t$.

Under the structural factorization constraint (See Appendix A.1 for more on structural VMP).

$$q(\mathit{y},\mathit{x},\mathbf{\theta },\gamma )=q(\mathit{y},\mathit{x})q\left(\mathbf{\theta }\right)q\left(\gamma \right),$$

(31)

and consistency constraints

$$q\left(\mathit{y}\right)=\int q(\mathit{y},\mathit{x})\mathrm{d}\mathit{x},q\left(\mathit{x}\right)=\int q(\mathit{y},\mathit{x})\mathrm{d}\mathit{y}$$

(32)

the marginals $q\left(\mathbf{\theta }\right)$, $q\left(\mathit{x}\right)$, $q\left(\mathit{y}\right)$ and $q\left(\gamma \right)$ can be obtained from the minimisation of the composite-AR free energy functional

$$\begin{array}{cc}\hfill F_{\mathrm{AR}}\left[q\right]& =\int q(\mathit{y},\mathit{x})q\left(\mathbf{\theta }\right)q\left(\gamma \right)log\frac{\stackrel{\mathrm{posterior}}{\overbrace{q(\mathit{y},\mathit{x})q\left(\mathbf{\theta }\right)q\left(\gamma \right)}}}{\underset{\mathrm{AR}\mathrm{node}}{\underbrace{f_{\mathrm{AR}}(\mathit{y},\mathit{x},\mathbf{\theta },\gamma )}}}\mathrm{d}\mathit{y}\mathrm{d}\mathit{x}\mathrm{d}\mathbf{\theta }\mathrm{d}\gamma .\hfill \end{array}$$

(33)

Recalling (25), we can write the minimizer of FE functional (33) with respect to $\mathbf{\theta }$ as

$$q\left(\mathbf{\theta }\right)\propto \overrightarrow{\nu }\left(\mathbf{\theta }\right)\overleftarrow{\nu }\left(\mathbf{\theta }\right)$$

(34)

where $q\left(\mathbf{\theta }\right)$ is associated with the incoming message to AR node and $\overrightarrow{\nu }\left(\mathbf{\theta }\right)$ is a variational outgoing message. Hence, the outgoing message from the AR node toward $\mathbf{\theta }$ can be written as

$$\begin{array}{cc}\hfill \overrightarrow{\nu }\left(\mathbf{\theta }\right)& \propto exp\left({\mathbb{E}}_{q(\mathit{y},\mathit{x})q\left({\mathbf{\theta }}_t\right)q\left(\gamma \right)}log\left[\mathcal{N}\left(\mathit{y}|\mathit{Ax},\mathit{V}\right)\right]\right)\hfill \end{array}$$

(35)

In Appendix A we work out a closed-form solution for this and all other update rules plus an evaluation of free energy for the composite AR node. The results are reported in Table 1. With these rules in hand, the composite AR node can be plugged into any factor graph and take part in a message passing-based free energy minimization process.

6. Experiments

In this section, we first verify the proposed methodology by a simulation of the proposed TVAR model on synthetic data, followed by validation experiments on two real world problems. We implemented all derived message passing rules in the open source Julia package ForneyLab.jl [28]. The code for the experiments and for the AR node can be found in public Github repositories. (https://github.com/biaslab/TVAR_FFG, accessed on 27 May 2021, https://github.com/biaslab/LAR, accessed on 27 May 2021) We used the following computer configuration to run the experiments. Operation system: macOS Big Sur, Processor: 2,7 GHz Quad-Core Intel Core $i7$, RAM: 16 GB.

6.1. Verification on a Synthetic Data Set

To verify the proposed TVAR inference methods, we synthesized data from two generative models $m_1$ and $m_2$, as follows:

$$\begin{array}{cc}\hfill {\mathbf{\theta }}_t& \sim \left\{\begin{array}{cc}\delta ({\mathbf{\theta }}_t-{\mathbf{\theta }}_{t-1})\hfill & \mathrm{if}m=m_1\hfill \\ \mathcal{N}({\mathbf{\theta }}_{t-1},\omega {\mathrm{I}}_M)\hfill & \mathrm{if}m=m_2\hfill \end{array}\right.\hfill \end{array}$$

(36a)

$$\begin{array}{cc}\hfill {\mathit{x}}_t& \sim \mathcal{N}\left(A\left({\mathbf{\theta }}_t\right){\mathit{x}}_{t-1},V\left(\gamma \right)\right)\hfill \end{array}$$

(36b)

$$\begin{array}{cc}\hfill y_t& \sim \mathcal{N}({\mathit{c}}^T{\mathit{x}}_t,\tau )\hfill \end{array}$$

(36c)

with priors

$$\begin{array}{cc}\hfill p(M=k)& =\prod _{k=1}^{10}0.1^{[M=k]}\hfill \end{array}$$

(37a)

$$\begin{array}{cc}\hfill {\mathbf{\theta }}_0& \sim \left\{\begin{array}{cc}\mathcal{N}(\mathbf{0},\mathit{I})\hfill & \mathrm{if}m=m_1\hfill \\ \mathcal{N}(\mathbf{0},1e12\mathit{I})\hfill & \mathrm{if}m=m_2\hfill \end{array}\right.\hfill \end{array}$$

(37b)

$$\begin{array}{cc}\hfill {\mathit{x}}_0& \sim \mathcal{N}(\mathbf{0},1e12\mathit{I})\hfill \end{array}$$

(37c)

$$\begin{array}{cc}\hfill \gamma & \sim \mathsf{\Gamma }(1.0,1e-5)\hfill \end{array}$$

(37d)

$$\begin{array}{cc}\hfill \tau & =1.0\hfill \end{array}$$

(37e)

$$\begin{array}{cc}\hfill \omega & =0.01\hfill \end{array}$$

(37f)

where M is the number of AR coefficients. Although these models differ only with respect to the properties of the AR coefficients $\mathbf{\theta }$, this variation has an important influence on the data generative process. The first model $m_1$ specifies a stationary AR process, since $\delta ({\mathbf{\theta }}_t-{\mathbf{\theta }}_{t-1})$ in (36a) indicates that $\theta$ is not time-varying in $m_1$. The second model $m_2$ represents a proper TVAR process as the prior evolution of the AR coefficients follows a random walk. One-time segment FFGs corresponding to the Equation (36) are depicted in Figure 4.

Figure 4. Forney-style Factor Graphs corresponding to Equation (36). (Left) model $m_1$. (Right) model $m_2$.

For each model, we generated a data set of 100 different time series, each of length 100 (so we have $2\times 100\times 100$ data points). For each time series, as indicated by (37a), the AR order M of the generative process was randomly drawn from the set $\{1,2,\dots ,10\}$. We used rather non-informative/broad priors for states and parameters for both models, see (37). This was done to ensure that the effect of the prior distributions is negligible relative to the information in the data set.

These time series were used in the following experiments. We selected two recognition models $m_1$ and $m_2$ with the same specifications as were used for generating the data set. The recognition models were trained on time series that were generated by models with the same AR order.

We proceeded by computing the quantities $q\left({\mathit{x}}_{1:T}\right|{\mathit{y}}_{1:T})$, $q\left({\mathbf{\theta }}_{1:T}\right|{\mathit{y}}_{1:T})$, $q\left(\gamma \right|{\mathit{y}}_{1:T})$ and $F\left[q({\mathit{z}}_{t-1},{\mathit{z}}_t|{\mathit{y}}_{1:T})\right]$ (where $\mathit{z}$ comprises all latent states and parameters) for both models, following the proposed rules from Table 1.

As a verification check, we first want to ensure that inference recovers the hidden states $x_t$ for each $t\in (1,2,\cdots ,100)$. Secondly, we want to verify the convergence of FE. As we have not used any approximations along the derivations of variational messages, we expect a smoothly decreasing curve for FE until convergence. The results of the verification stage are highlighted for a typical case in Figure 5. The figure confirms that states $x_t$ are accurately tracked and that a sliding average of the AR coefficients ${\mathbf{\theta }}_t$ is also nicely tracked. Figure 5 also indicates that the FE uniformly decreases towards lower values as we spend more computational power.

Figure 5. Verification results. The solid line corresponds to the value of the latent (hidden) states in the generative processes. The dashed line corresponds to the expected mean value of the posterior estimates of hidden states $q(·|{\mathit{y}}_{1:100})$ in the recognition models. The shadowed regions corresponds to one standard deviation of the posteriors in the recognition models below and above the estimated mean. The top two plots show inference results for the coefficients ${\mathbf{\theta }}_t$ (top-left) and states $x_t$ (top-right) of TVAR(1) (model $m_2$, AR order $M=1$) for time series ♯10. (bottom-left) State trajectory $q\left(x_t\right|{\mathit{y}}_{1:100})$ model $m_1$, AR order $M=1$ on time series ♯99. (Bottom-right) Evolution of FE for $m_1$ (AR) and $m_2$ (TVAR), averaged over their corresponding time series. The iteration number at the abscissa steps through a single marginal update for all edges in the graph.

We note that the FE score by itself does not explain whether the model is good or not, but it serves as a good measure for model comparison. In the following subsection, we demonstrate how FE scores can be used for model selection.

6.2. Temperature Modeling

AR models are well-known for predicting different weather conditions such as wind, temperature, precipitation, etc. Here, we will revisit the problem of modeling daily temperature. We used a data set of daily minimum temperatures (in °C) in Melbourne, Australia, 1981–1990 (3287 days) (https://www.kaggle.com/paulbrabban/daily-minimum-temperatures-in-melbourne, accessed on 27 May 2021). We then corrupted the data set by adding random noise sampled from $\mathcal{N}(0,10.0)$ to the actual temperatures. A fragment of the time-series is depicted in Figure 6.

Figure 6. Temperature time-series from days 2000 to 2200. Crosses denote the thermometer readings plus added noise. The solid line corresponds to the latent (hidden) daily temperature.

To estimate the actual temperature based on past noisy observations by computing $q\left({\mathit{x}}_t\right|{\mathit{y}}_{1:t})$, we use a TVAR model with measurement noise to simulate uncertainty about corrupted observations. The model is specified by the following equation set

$$\begin{array}{cc}\hfill {\mathbf{\theta }}_t& \sim \mathcal{N}({\mathbf{\theta }}_{t-1},{\mathrm{I}}_M)\hfill \end{array}$$

(38a)

$$\begin{array}{cc}\hfill {\mathit{x}}_t& \sim \mathcal{N}\left(A\left({\mathbf{\theta }}_t\right){\mathit{x}}_{t-1}+\mathit{c}\eta ,V\left(\gamma \right)\right)\hfill \end{array}$$

(38b)

$$\begin{array}{cc}\hfill y_t& \sim \mathcal{N}({\mathit{c}}^{⊺}{\mathit{x}}_t,\tau )\hfill \end{array}$$

(38c)

with priors

$$\begin{array}{cc}\hfill {\mathbf{\theta }}_0\sim \mathcal{N}(\mathbf{0},& \mathit{I}){\mathit{x}}_0\sim \mathcal{N}(\mathbf{0},\mathit{I})\eta \sim \mathcal{N}(0.0,10.0)\hfill \end{array}$$

(39a)

$$\begin{array}{cc}\hfill & \gamma \sim \mathsf{\Gamma }(1.0,1.0)\tau \sim \mathsf{\Gamma }(0.1,1.0)\hfill \end{array}$$

(39b)

Since the temperature data is not centered around 0 °C, we added a bias term $\eta$ to the state ${\mathit{x}}_t$. The corresponding FFG is depicted in Figure 7.

Figure 7. One time segment of a Forney-style factor graph (FFG) for the TVAR model in the temperature modeling task (38).

Note that we put a Gamma prior on the measurement noise precision $\tau$, meaning that we are uncertain about the size of the error of the thermometer reading. The inference task for the model is computing $q\left({\mathit{x}}_t\right|{\mathit{y}}_{1:t})$, in other words, we track the states based only on past data. Of course, after training, we could use the model for temperature prediction by tracking $q\left({\mathit{x}}_{t+k}\right|{\mathit{y}}_{1:t})$ for $k\ge 1$. We compare the performance of four TVAR models with AR orders $M=\{1,2,3,4\}$. To choose the best model, we computed the average FE score for each TVAR(M) model.

Figure 8 shows that on average TVAR(3) outperforms its competitors. The complexity vs accuracy decomposition (23) of FE explains why a lower order model may outperform higher order models. TVAR(4) maybe as accurate or more accurate than TVAR(3) but the increase in accuracy is more than offset by the increase in complexity. For the lower order models, it is the other way around: they are less complex and involve fewer computations than TVAR(3), but the loss in model complexity leads to too much loss in data modeling accuracy. Overall, TVAR(3) is the best model for this data set. Practically, we always favor the model that features the lowest FE score. In the next subsection we will use this technique (scoring FE) for online model selection.

Figure 8. (Left) Comparison of four TVAR(M) models for the temperature filtering problem. Bars correspond to the averaged (over 3287 days) FE score for each model. (Right) Inference example of the best performing model (TVAR(3)). Crosses denote the thermometer reading plus added noise. The solid line corresponds to the latent (hidden) daily temperature. The dashed line corresponds to the mean of the posterior estimates of hidden temperature and the shadowed region corresponds to one standard deviation below and above the estimated temperature.

6.3. Single-Channel Speech Enhancement

Single-channel speech enhancement (SCSE) is a well-known challenging task that aims to enhance noisy speech signals that were recorded by a single microphone. In single microphone recordings, we cannot use any spatial information that is commonly used in beamforming applications. Much work has been done to solve the SCSE task, ranging from Wiener filter-inspired signal processing techniques [35,36] to deep learning neural networks [37]. In this paper, we use data from the speech corpus (NOIZEUS) (https://ecs.utdallas.edu/loizou/speech/noizeus/, accessed on 27 May 2021) [38] and corrupted clean speech signals with white Gaussian noise, leading to a signal-to-noise ratio (SNR)

$$\mathrm{SNR}({\mathbf{s}}_{1:T},{\mathit{y}}_{1:T})=10{log}_{10}\left[\frac{{\sum }_t^T{s}_t^2}{{\sum }_t^T{(s_t-y_t)}^2}\right]\approx 13.36\mathrm{dB}$$

(40)

where ${\mathbf{s}}_{1:T}=(s_1,\cdots ,s_T)$ and ${\mathit{y}}_{1:T}=(y_1,\cdots ,y_T)$ are clean and corrupted speech signals. $s_t$ is a speech signal at time t and T is the length of the signal.

Historically, AR models have shown to perform well for modeling speech signals in the time (waveform) domain [39,40]. Despite the fact that speech is a highly nonstationary signal, we may assume it to be stationary within short time intervals (frames) of about 10 [ms] each [41]. Since voiced, unvoiced and silence frames have very different characteristics, we used 5 different models (a random walk model (RW), AR(1), AR(2), TVAR(1) and TVAR(2)) for each frame of 10 [ms] with 2.5 [ms] overlap. Given a sampling frequency of 8 [kHZ], each frame results into 80 samples with 20 samples overlap. The AR and TVAR models were specified by Equation (36). For each frame, we evaluated the model performance by minimized FE and selected the model with minimal FE score. We used identical prior parameters for all models where possible. To recover the speech signal we computed the mean values of $q\left({\mathit{x}}_t\right|y_{1:T})$ of the selected model for each frame. The SNR gain of this SCSE system was

$$\mathrm{SNR}({\mathbf{s}}_{1:T},{\mathit{x}}_{1:T})-\mathrm{SNR}({\mathbf{s}}_{1:T},{\mathit{y}}_{1:T})\approx 3.7\mathrm{dB}.$$

(41)

Figure 9 show the spectrograms of the clean, noisy and filtered signal respectively.

Figure 9. Spectrogram of recovered speech signal in the experiment of Section 6.3.

Next, we analyze the inference results in a bit more detail. Table 2 shows the percentage of winning models for each frame based on the free energy score.

Table 2. Percentage of preferred models (based on FE scores) for all frames on the speech enhancement task.

	RW	AR(1)	AR(2)	TVAR(1)	TVAR(2)
Ratio	$32.2$%	$54.3$%	$10.7$%	$1.2$%	$0.5$%

As we can see, for more than 30% of all frames, the random walk model performs best. This happens mostly because for a silent frame the AR model gets penalized by its complexity term. We recognize that in about 90% of the frames the best models are AR(1) and RW. On the other hand, for the frames where the speech signal transitions from silent or unvoiced to voiced, these fixed models start to fail and the time-varying AR models perform better. This effect can be seen in Figure 10.

Figure 10. (Top) (Top-left) Inference by TVAR(2) for the segment 293. (Top-right) Inference by RW for the segment 293. Note how the TVAR model is able to follow the transitions at the end of the frame, while the RW cannot adapt within one frame. (Bottom) FE scores from segment 291 to 295. TVAR(2) wins frame 293 as it has the lowest FE score.

Figure 11 shows the performance of the AR(2) and RW models on a frame with a voiced speech signal. For this case, the AR(2) model performs better.

Figure 11. Comparison of AR(2) and RW models for a voiced signal frame. (Top-left) Inference by AR(2) for the segment 208. (Top-right) Inference by RW for the segment 208. (Bottom) FE scores from segment 206 to 210. The AR(2) model wins frame 208.

Finally, Figure 12 shows how the TVAR(2) model compares to the RW model on one of the unvoiced/silence frames. While the estimates of TVAR(2) appear to be more accurate, it pays a bigger “price” for the model complexity term in the FE score and the RW model wins the frame.

Figure 12. Comparison of TVAR(2) and RW models for an unvoiced/silence frame. (Top-left) Inference by TVAR(2) for the frame 62. (Top-right) Inference by RW for the frame 62. (Bottom) FE scores from segment 60 to 64. The RW model scores best on frame 62 due to its low complexity.

7. Discussion

We have introduced a TVAR model that includes efficient joint variational Bayesian tracking of states, parameters and free energy. The system can be used as a plug-in module in factor graph-based representations of other models. At several points in this paper, we have made some design decisions that we shortly review here.

While FE computation for the AR node provides a convenient performance criterion for model selection, we noticed in the speech enhancement simulation that separate FE tracking for each candidate model leads to a large computational overhead. There are ways to improve the model selection process that we used in the speech enhancement simulation. One way is to consider a mixture model of candidate models and track the posterior over the mixture coefficients [42]. Alternatively, a very cheap method for online Bayesian model selection may be the recently developed Bayesian Model Reduction (BMR) method [43]. The BMR method is based on a generalization of the Savage-Dickey Density Ratio and supports tracking of free energy of multiple nested models with almost no computational overhead. Both methods seem to integrate well with a factor graph representation and we plan to study this issue in future work.

In this paper, the posterior factorization (31) supports the modeling of temporal dependencies between input and output of the AR node in the posterior. Technically, (31) corresponds to a structural VMP assumption, in contrast to the more constrained mean-field VMP algorithm that would be based on $q\left(\mathit{z}\right)={\prod }_i{q}_i\left(z_i\right)$, where $\mathit{z}$ is the set of all latent variables [44]. We could have also worked out alternative update rules for the assumption of a joint factorization of precision $\gamma$ and AR coefficients $\mathbf{\theta }$. In that case, the prior (incoming message $\overrightarrow{\nu }(\mathbf{\theta },\gamma )$ to AR node) would be in the form of a Normal-Gamma distribution. While any of these these assumptions are technically valid, each choice accepts a different trade-off in the accuracy vs. complexity space. We review structural VMP in Appendix A.1.

In the temperature modelling task, we added some additional random variables (bias, measurement noise precision). To avoid identifiability issues, in (38a) we fixed the covariance matrix of the time-varying AR coefficient to the identity matrix. In principle, this constraint can be relaxed. For example, an Inverse-Wishart prior distribution can be added to the covariance matrix.

In our speech enhancement experiments in Section 6.3, we assume that the measurement noise variance is known. In a real-world scenario, this information is usually not accessible. However, online tracking of measurement noise or other (hyper-)parameters is usually not a difficult extension when the process is simulated in a factor graph toolbox such as ForneyLab [28]. If so desired, we could add a prior on the measurement noise variance and track the posterior. The online free energy criterion (23) can be used to determine if the additional computational load (complexity) of Bayesian tracking of the variance parameter has been compensated by the increase in modeling accuracy.

The realization of the TVAR model in ForneyLab comes with some limitations. For large smoothing problems (say, >1000 data points), the computational load of message passing in ForneyLab becomes too heavy for a standard laptop (as was used in the paper). Consequently, in the current implementation it is difficult to employ the AR node for processing large time series on a standard laptop. To circumvent this issue, when using ForneyLab, one can combine filtering and smoothing solutions into a batch learning procedure. In future work we plan to remedy this issue by some ForneyLab refactoring work. Additionally, the implemented AR node does not provide a closed-form update rule for the marginal distribution when the probability distribution types of the incoming messages (priors) are different from the ones used in our work. Fortunately, ForneyLab supports resorting to (slower) sampling-based update rules when closed-form update rules are not available.

8. Conclusions

We presented a variational message passing approach to tracking states and parameters in latent TVAR models. The required update rules have been summarized and implemented in the factor graph package ForneyLab.jl, thus making transparent usage of TVAR factors available in freely definable stochastic dynamical systems. Aside from VMP update rules, we derived a closed-form expression for the variational free energy (FE) of an AR factor. Free Energy can be used as a proxy for Bayesian model evidence and as such allows for model performance comparisons between the TVAR models and alternative structures. Owing to the locality and modularity of the FFG framework, we demonstrated how AR nodes can be applied as plug-in modules in various dynamic models. We verified the correctness of the rules on a synthetic data set and applied the proposed TVAR model to a few relatively simple but different real-world problems. In future work, we plan to extend the current factor graph-based framework to efficient and transparent tracking of AR model order and to online model comparison and selection with alternative models.

Appendix A.1. Structural Variational Message Passing

The message update rule (24) implies a mean-field factorization, meaning that all variables represented by edges around the factor node f are independent. In this paper, we impose a structural dependence between states. To illustrate how structured VMP works, let us consider the example depicted in Figure A2.

Figure A2. A node $f(x,y,z)$ representing an arbitrary joint distribution. Arrows above the messages $\nu (·)$ indicate the direction (incoming or outgoing).

Suppose that we constrain the joint posterior (A1) as

$$q(x,y,z)=q(x,y)q\left(z\right)$$

(A1)

The message passing algorithm for updating the marginal posteriors $q^{*}(x,y)$ and $q^{*}\left(z\right)$ can now be executed as follows:

(1)

compute outgoing messages $\overrightarrow{\nu }\left(y\right)$, $\overleftarrow{\nu }\left(x\right)$:

$$\begin{array}{cc}\hfill \overrightarrow{\nu }\left(y\right)& \propto \int \overrightarrow{\nu }\left(x\right)exp\left(\int q\left(z\right)log\left[f(x,y,z)\right]\mathrm{d}z\right)\mathrm{d}x\hfill \end{array}$$

(A2a)

$$\begin{array}{cc}\hfill \overleftarrow{\nu }\left(x\right)& \propto \int \overleftarrow{\nu }\left(y\right)exp\left(\int q\left(z\right)log\left[f(x,y,z)\right]\mathrm{d}z\right)\mathrm{d}y\hfill \end{array}$$

(A2b)

(2)

update joint posterior $q^{*}(x,y)$:

$$q^{*}(x,y)\propto \overrightarrow{\nu }\left(x\right)exp\left(\int q\left(z\right)logf(x,y,z)\mathrm{d}z\right)\overleftarrow{\nu }\left(y\right)$$

(A3)

(3)

compute the outgoing message $\overrightarrow{\nu }\left(z\right)$:

$$\overrightarrow{\nu }\left(z\right)\propto exp\left(\int q^{*}(x,y)logf(x,y,z)\mathrm{d}x\mathrm{d}y\right),$$

(A4)

(4)

update posterior $q^{*}\left(z\right)$:

$$q^{*}\left(z\right)\propto \overrightarrow{\nu }\left(z\right)\overleftarrow{\nu }\left(z\right)$$

(A5)

Every marginal update rule (Equations (25), (A3) and (A5)) corresponds to a coordinate descent step on the variational free energy, and therefore the free energy is guaranteed to converge to a local minimum.

Appendix A.2. Auxiliary Node Function

Before obtaining the update messages for TVAR we need to evaluate the auxiliary node function $\tilde{f}(\mathit{x},\mathit{y})\propto exp\left\{{\mathbb{E}}_{q\left(\gamma \right)q\left(\mathbf{\theta }\right)}log\left[f(\mathit{y}\mathit{x},\mathbf{\theta },\gamma )\right]\right\}$. We also need to address the issue of invertability of the covariance matrix V. To tackle this problem, we assume $ϵ>0$, ${ϵ}^2\approx 0$ which allows us to introduce matrix $\mathit{W}={\mathit{V}}^{-1}$$({\mathit{W}}^{-1}\mathit{V}={\mathit{V}}^{-1}\mathit{W}=I)$.

$$\begin{array}{c}\hfill \mathit{V}=\left[\begin{array}{ccccc}{\gamma }^{-1}& 0& 0& \cdots & 0\\ 0& ϵ& 0& \cdots & ⋮\\ 0& 0& ϵ& \cdots & ⋮\\ ⋮& ⋮& \ddots & \ddots & ⋮\end{array}\right]\end{array}$$

$$\begin{array}{cc}\hfill log\tilde{f}(\mathit{x},\mathit{y})& ={\mathbb{E}}_{q\left(\gamma \right)q\left(\mathbf{\theta }\right)}logf(\mathit{y}\mathit{x},\mathbf{\theta },\gamma )+const\hfill \\ \hfill & =\frac{1}{2}{\mathbb{E}}_{q\left(\gamma \right)}\left[log\left|\mathit{W}\right|\right]-\frac{1}{2}{\mathbb{E}}_{q\left(\gamma \right)q\left(\mathbf{\theta }\right)}\left[{\left(\mathit{y}-\mathit{Ax}\right)}^{\top }\mathit{W}\left(\mathit{y}-\mathit{Ax}\right)\right]+const\hfill \\ \hfill & =-\frac{1}{2}{\mathbb{E}}_{q\left(\gamma \right)q\left(\mathbf{\theta }\right)}\left[\mathrm{tr}\left(\mathit{W}\left(\mathit{y}-\mathit{Ax}\right){\left(\mathit{y}-\mathit{Ax}\right)}^{\top }\right)\right]+const\hfill \\ \hfill & =-\frac{1}{2}\mathrm{tr}\left(m_{\mathit{W}}{\mathbb{E}}_{q\left(\mathbf{\theta }\right)}\left[\left(\mathit{y}-\mathit{Ax}\right){\left(\mathit{y}-\mathit{Ax}\right)}^{\top }\right]\right)+const\hfill \\ \hfill & =-\frac{1}{2}\mathrm{tr}\left(m_{\mathit{W}}\left({\mathit{yy}}^{\top }-m_{\mathit{A}}{\mathit{xy}}^{\top }-{\mathit{yx}}^{\top }{m}_{\mathit{A}}+{\mathbb{E}}_{q\left(\mathbf{\theta }\right)}\left[{\mathit{Axx}}^{\top }{\mathit{A}}^{\top }\right]\right)\right)\hfill \\ \hfill & +const\hfill \end{array}$$

We work out the expectation term inside the trace separately. To do this, we notice, that the product $\mathit{A}\mathit{x}$ can be separated in the shifting operator $\mathit{Sx}$ and the inner vector product ${\mathit{cx}}^{\top }\mathbf{\theta }$ in the following way:

$$\mathit{A}\mathit{x}=\mathit{Sx}+{\mathit{cx}}^{\top }\mathbf{\theta }=\underset{\mathit{A}\mathit{x}}{\underbrace{\left(\mathit{S}+\mathit{c}{\mathbf{\theta }}^{\top }\right)}}$$

(A6)

where

$$\begin{array}{c}\hfill \mathit{S}=\left[\begin{array}{c}{\mathbf{0}}^{\top }\\ {\mathit{I}}_{M-1}\mathbf{0}\end{array}\right]\mathit{c}={(1,0,\cdots ,0)}^{\top }\end{array}$$

$$\begin{array}{cc}\hfill & {\mathbb{E}}_{q\left(\mathbf{\theta }\right)}\left[{\mathit{Axx}}^{\top }{\mathit{A}}^{\top }\right]={\mathbb{E}}_{q\left(\mathbf{\theta }\right)}\left[\left(\mathit{Sx}+{\mathit{cx}}^{\top }\mathbf{\theta }\right){\left(\mathit{Sx}+{\mathit{cx}}^{\top }\mathbf{\theta }\right)}^{\top }\right]\hfill \\ \hfill & ={\mathbb{E}}_{q\left(\mathbf{\theta }\right)}\left[\mathit{Sx}{\left(\mathit{Sx}\right)}^{\top }+{\mathit{cx}}^{\top }\mathbf{\theta }{\left(\mathit{Sx}\right)}^{\top }+\mathit{Sx}({\mathit{cx}}^{\top }{\mathbf{\theta })}^{\top }+{\mathit{cx}}^{\top }\mathbf{\theta }{\mathbf{\theta }}^{\top }{\mathit{xc}}^{\top }\right]\hfill \\ \hfill & =\mathit{Sx}{\left(\mathit{Sx}\right)}^{\top }+{\mathit{cx}}^{\top }{m}_{\mathbf{\theta }}{\left(\mathit{Sx}\right)}^{\top }+\mathit{Sx}({\mathit{cx}}^{\top }{m}_{\mathbf{\theta }}{)}^{\top }+{\mathit{cx}}^{\top }\left[V_{\mathbf{\theta }}+m_{\mathbf{\theta }}{m}_{\mathbf{\theta }}^{\top }\right]{\mathit{xc}}^{\top }\hfill \\ \hfill & =\left(\mathit{Sx}+{\mathit{cx}}^{\top }{m}_{\mathbf{\theta }}\right){\left(\mathit{Sx}+{\mathit{cx}}^{\top }{m}_{\mathbf{\theta }}\right)}^{\top }+{\mathit{cx}}^{\top }{V}_{\mathbf{\theta }}{\mathit{xc}}^{\top }\hfill \\ \hfill & =m_{\mathit{A}}x{\left(m_{\mathit{A}}x\right)}^{\top }+{\mathit{cx}}^{\top }{V}_{\mathbf{\theta }}{\mathit{xc}}^{\top }\hfill \end{array}$$

Hence

$$\begin{array}{cc}\hfill & log\tilde{f}\left(\mathit{y},\mathit{x}\right)\hfill \\ \hfill & =-\frac{1}{2}\mathrm{tr}\left(m_{\mathit{W}}\left[{\mathit{yy}}^{\top }-m_{\mathit{A}}{\mathit{xy}}^{\top }-{\mathit{yx}}^{\top }{m}_{\mathit{A}}+m_{\mathit{A}}\mathit{x}{\left(m_{\mathit{A}}\mathit{x}\right)}^{\top }+{\mathit{cx}}^{\top }{V}_{\mathbf{\theta }}{\mathit{xc}}^{\top }\right]\right)\hfill \\ \hfill & +const\hfill \\ \hfill & =-\frac{1}{2}\left({\mathit{y}}^{\top }{m}_{\mathit{W}}\mathit{y}-{\mathit{y}}^{\top }{m}_{\mathit{W}}{m}_{\mathit{A}}\mathit{x}-{\left(m_{\mathit{A}}\mathit{x}\right)}^{\top }{m}_{\mathit{W}}\mathit{y}+{\left(m_{\mathit{A}}\mathit{x}\right)}^{\top }{m}_{\mathit{W}}{m}_{\mathit{A}}\mathit{x}\right)\hfill \\ \hfill & -\frac{m_{\gamma }}{2}{\mathit{x}}^{\top }{V}_{\mathbf{\theta }}\mathit{x}+const\hfill \\ \hfill & =-\frac{1}{2}{(\mathit{y}-m_{\mathit{A}}\mathit{x})}^{\top }{m}_{\mathit{W}}(\mathit{y}-m_{\mathit{A}}\mathit{x})-\frac{m_{\gamma }}{2}{\mathit{x}}^{\top }{V}_{\mathbf{\theta }}\mathit{x}+const\hfill \end{array}$$

We can write the auxiliary node function as

$$\begin{array}{c}\hfill \tilde{f}\left(\mathit{x},\mathit{y}\right)\propto \mathcal{N}\left(\mathit{y}\right|m_{\mathit{A}}\mathit{x},m_{\mathit{W}}^{-1})\mathcal{N}\left(\mathit{x}\right|\mathbf{0},{\left(m_{\gamma }{V}_{\theta }\right)}^{-1})\end{array}$$

(A7)

Appendix A.3. Update of Message to y

Owing Equation (A7),

$$\begin{array}{cc}\hfill \overrightarrow{\nu }\left(\mathit{y}\right)& \propto \int \overrightarrow{\nu }\left(\mathit{x}\right)\tilde{f}(\mathit{x},\mathit{y})\mathrm{d}\mathit{x}\hfill \\ \hfill & \propto \int \mathcal{N}\left(\mathit{x}\right|m_{\mathit{x}},V_{\mathit{x}})\mathcal{N}\left(\mathit{y}\right|m_{\mathit{A}}\mathit{x},m_{\mathit{W}}^{-1})\mathcal{N}\left(\mathit{x}\right|\mathbf{0},{\left(m_{\gamma }{V}_{\theta }\right)}^{-1})\mathrm{d}\mathit{x}\hfill \\ \hfill & \propto \int \mathcal{N}\left(\mathit{x}\right|{\mathsf{\Lambda }}^{-1}\mathit{z},{\mathsf{\Lambda }}^{-1})\mathcal{N}\left(\mathit{y}\right|m_{\mathit{A}}\mathit{x},m_{\mathit{W}}^{-1})\mathrm{d}\mathit{x}\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill & \mathsf{\Lambda }=V_{\mathit{x}}^{-1}+m_{\gamma }{V}_{\mathbf{\theta }}\hfill \\ \hfill & \mathit{z}=V_{\mathit{x}}^{-1}{m}_{\mathit{x}}\hfill \end{array}$$

In this way, the message $\overrightarrow{\nu }\left(\mathit{y}\right)$

$$\begin{array}{cc}\hfill \overrightarrow{\nu }\left(\mathit{y}\right)& \propto \int \mathcal{N}\left(\mathit{x}\right|{\mathsf{\Lambda }}^{-1}\mathit{z},{\mathsf{\Lambda }}^{-1})\mathcal{N}\left(\mathit{y}\right|m_{\mathit{A}}\mathit{x},m_{\mathit{W}}^{-1})\mathrm{d}\mathit{x}\hfill \\ & \propto \mathcal{N}\left(\mathit{y}|m_{\mathit{A}}{(V_{\mathit{x}}^{-1}+m_{\gamma }{V}_{\mathbf{\theta }})}^{-1}{V}_{\mathit{x}}^{-1}{m}_{\mathit{x}},m_{\mathit{A}}{(V_{\mathit{x}}^{-1}+m_{\gamma }{V}_{\mathbf{\theta }})}^{-1}{m}_{\mathit{A}}^{\top }+m_{\mathit{V}}\right)\hfill \end{array}$$

Appendix A.4. Update of Message to x

Owing Equation (A7),

$$\begin{array}{cc}\hfill \overleftarrow{\nu }\left(\mathit{x}\right)& \propto \int \overleftarrow{\nu }\left(\mathit{y}\right)\tilde{f}(\mathit{x},\mathit{y})\mathrm{d}\mathit{y}\hfill \\ \hfill & \propto \int \mathcal{N}\left(\mathit{y}\right|m_{\mathit{y}},V_{\mathit{y}})\mathcal{N}\left(\mathit{y}\right|m_{\mathit{A}}\mathit{x},m_{\mathit{W}}^{-1})\mathcal{N}\left(\mathit{x}\right|\mathbf{0},{\left(m_{\gamma }{V}_{\theta }\right)}^{-1})\mathrm{d}\mathit{y}\hfill \end{array}$$

Let us consider the log of $\mathcal{N}\left(\mathit{y}\right|m_{\mathit{A}}\mathit{x},m_{\mathit{W}}^{-1})$:

$$\begin{array}{cc}\hfill log\left[\mathcal{N}\left(\mathit{y}\right|m_{\mathit{A}}\mathit{x},m_{\mathit{W}}^{-1})\right]& ={(\mathit{y}-m_{\mathit{A}}\mathit{x})}^{\top }{m}_{\mathit{W}}(\mathit{y}-m_{\mathit{A}}\mathit{x})+const\hfill \\ \hfill & ={(-m_{\mathit{A}}^{-1}\mathit{y}+\mathit{x})}^{\top }{m}_{\mathit{A}}^{\top }{m}_{\mathit{W}}{m}_{\mathit{A}}(-m_{\mathit{A}}^{-1}\mathit{y}+\mathit{x})+const\hfill \end{array}$$

Which yields,

$$\begin{array}{c}\hfill \mathcal{N}\left(\mathit{y}\right|m_{\mathit{A}}\mathit{x},m_{\mathit{W}}^{-1})\propto \mathcal{N}\left(\mathit{x}\right|m_{\mathit{A}}^{-1}\mathit{y},{\left(m_{\mathit{A}}^{\top }{m}_{\mathit{W}}{m}_{\mathit{A}}\right)}^{-1})\end{array}$$

(A8)

Therefore,

$$\begin{array}{cc}\hfill \overleftarrow{\nu }\left(\mathit{x}\right)& \propto \int \mathcal{N}\left(\mathit{y}\right|m_{\mathit{y}},V_{\mathit{y}})\mathcal{N}\left(\mathit{x}\right|m_{\mathit{A}}^{-1}\mathit{y},{\left(m_{\mathit{A}}^{\top }{m}_{\mathit{W}}{m}_{\mathit{A}}\right)}^{-1})\mathcal{N}\left(\mathit{x}\right|\mathbf{0},{\left(m_{\gamma }{V}_{\theta }\right)}^{-1})\mathrm{d}\mathit{y}\hfill \\ \hfill & \propto \mathcal{N}\left(\mathit{x}\right|\mathbf{0},{\left(m_{\gamma }{V}_{\theta }\right)}^{-1})\int \mathcal{N}\left(\mathit{y}\right|m_{\mathit{y}},V_{\mathit{y}})\mathcal{N}\left(\mathit{x}\right|m_{\mathit{A}}^{-1}\mathit{y},{\left(m_{\mathit{A}}^{\top }{m}_{\mathit{W}}{m}_{\mathit{A}}\right)}^{-1})\mathrm{d}\mathit{y}\hfill \\ \hfill & \propto \mathcal{N}\left(\mathit{x}\right|\mathbf{0},{\left(m_{\gamma }{V}_{\theta }\right)}^{-1})\mathcal{N}\left(\mathit{x}\right|m_{\mathit{A}}^{-1}{m}_{\mathit{y}},m_{\mathit{A}}^{-1}{V}_{\mathit{y}}{m}_{\mathit{A}}^{-\top }+{\left(m_{\mathit{A}}^{\top }{m}_{\mathit{W}}{m}_{\mathit{A}}\right)}^{-1})\hfill \\ \hfill & \propto \mathcal{N}\left(\mathit{x}\right|\mathbf{0},{\left(m_{\gamma }{V}_{\theta }\right)}^{-1})\mathcal{N}\left(\mathit{x}\right|m_{\mathit{A}}^{-1}{m}_{\mathit{y}},m_{\mathit{A}}^{-1}(V_{\mathit{y}}+m_{\mathit{V}})m_{\mathit{A}}^{-\top })\hfill \\ \hfill & \propto \mathcal{N}\left(\mathit{x}|{\mathsf{\Lambda }}^{-1}\mathit{z},{\mathsf{\Lambda }}^{-1}\right)\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill & \mathsf{\Lambda }=m_{\mathit{A}}^{\top }{\left(V_{\mathit{y}}+m_{\mathit{V}}\right)}^{-1}{m}_{\mathit{A}}+m_{\gamma }{V}_{\theta }\hfill \\ \hfill & \mathit{z}=m_{\mathit{A}}^{\top }{\left(V_{\mathit{y}}+m_{\mathit{V}}\right)}^{-1}{m}_{\mathit{y}}\hfill \end{array}$$

Appendix A.5. Update of Message to θ

The outgoing variational message to $\mathbf{\theta }$ is defined as

$$\begin{array}{c}\hfill \overleftarrow{\nu }\left(\mathbf{\theta }\right)\propto exp\left\{{\mathbb{E}}_{q(\mathit{x},\mathit{y})q\left(\gamma \right)}logf(\mathit{y}\mathit{x},\mathbf{\theta },\gamma )\right\}\end{array}$$

Instead of working out $\overleftarrow{\nu }\left(\mathbf{\theta }\right)$, we will work with corresponding log message

$$\begin{array}{cc}\hfill log\overleftarrow{\nu }\left(\mathbf{\theta }\right)& ={\mathbb{E}}_{q(\mathit{x},\mathit{y})q\left(\gamma \right)}\left[{log\left|\mathit{W}\right|}^{\frac{1}{2}}-\frac{1}{2}\left({(\mathit{y}-\mathit{Ax})}^{\top }\mathit{W}(\mathit{y}-\mathit{Ax})\right)\right]+const\hfill \\ \hfill & =-\frac{1}{2}\mathrm{tr}\left(m_{\mathit{W}}{\mathbb{E}}_{q\left(\mathit{x},\mathit{y}\right)}\left[\mathit{y}{\mathit{y}}^{\top }-\mathit{A}\mathit{x}{\mathit{y}}^{\top }-\mathit{y}{\left(\mathit{A}\mathit{x}\right)}^{\top }+\mathit{A}\mathit{x}{\left(\mathit{A}\mathit{x}\right)}^{\top }\right]\right)+const\hfill \\ \hfill & =-\frac{1}{2}\mathrm{tr}\left(m_{\mathit{W}}{\mathbb{E}}_{q\left(\mathit{x},\mathit{y}\right)}\left[-\mathit{A}\mathit{x}{\mathit{y}}^{\top }-\mathit{y}{\left(\mathit{A}\mathit{x}\right)}^{\top }+\mathit{A}\mathit{x}{\left(\mathit{A}\mathit{x}\right)}^{\top }\right]\right)+const\hfill \\ \hfill & =-\frac{1}{2}\mathrm{tr}\left(m_{\mathit{W}}{\mathbb{E}}_{q(\mathit{x},\mathit{y})}\left[-(\mathit{S}\mathit{x}+\mathit{c}{\mathit{x}}^{\top }\mathbf{\theta }){\mathit{y}}^{\top }-\mathit{y}{(\mathit{S}\mathit{x}+\mathit{c}{\mathit{x}}^{\top }\mathbf{\theta })}^{\top }\right]\right)\hfill \\ \hfill & -\frac{1}{2}\mathrm{tr}\left(m_{\mathit{W}}{\mathbb{E}}_{q(\mathit{x},\mathit{y})}(\mathit{S}\mathit{x}+\mathit{c}{\mathit{x}}^{\top }\mathbf{\theta }){(\mathit{S}\mathit{x}+\mathit{c}{\mathit{x}}^{\top }\mathbf{\theta })}^{\top }\right)+const\hfill \end{array}$$

To proceed further, we recall one useful property

$${\mathit{S}}^{\top }\Sigma \mathit{c}=\mathbf{0}{\mathit{c}}^{\top }\Sigma \mathit{S}={\mathbf{0}}^{\top }$$

where $\Sigma$ is an arbitrary diagonal matrix. Now, let us work out the following term

$$\begin{array}{cc}\hfill & \mathrm{tr}\left(m_{\mathit{W}}(\mathit{S}\mathit{x}+\mathit{c}{\mathit{x}}^{\top }\mathbf{\theta }){(\mathit{S}\mathit{x}+\mathit{c}{\mathit{x}}^{\top }\mathbf{\theta })}^{\top }\right)\hfill \\ & =\mathrm{tr}\left(m_{\mathit{W}}\left[\mathit{S}\mathit{x}{\mathit{x}}^{\top }{\mathit{S}}^{\top }+\mathit{S}\mathit{x}{\mathbf{\theta }}^{\top }\mathit{x}{\mathit{c}}^{\top }+\mathit{c}{\mathit{x}}^{\top }\mathbf{\theta }{\mathit{x}}^{\top }{\mathit{S}}^{\top }+\mathit{c}{\mathit{x}}^{\top }\mathbf{\theta }{\mathbf{\theta }}^{\top }\mathit{x}{\mathit{c}}^{\top }\right]\right)\hfill \\ \hfill & =\left[{\left(\mathit{S}\mathit{x}\right)}^{\top }{m}_{\mathit{W}}\mathit{S}\mathit{x}+\underset{{\mathbf{0}}^{\top }}{\underbrace{{\mathit{c}}^{\top }{m}_{\mathit{W}}\mathit{S}\mathit{x}{\mathbf{\theta }}^{\top }\mathit{x}}}+\underset{\mathbf{0}}{\underbrace{{\mathit{S}}^{\top }{m}_{\mathit{W}}\mathit{c}{\mathit{x}}^{\top }\mathbf{\theta }{\mathit{x}}^{\top }}}+{\mathit{c}}^{\top }{m}_{\mathit{W}}\mathit{c}{\mathit{x}}^{\top }\mathbf{\theta }{\mathbf{\theta }}^{\top }\mathit{x}\right]\hfill \\ \hfill & =\mathrm{tr}\left(m_{\mathit{W}}\left[\mathit{S}\mathit{x}{\mathit{x}}^{\top }{\mathit{S}}^{\top }+\mathit{c}{\mathit{x}}^{\top }\mathbf{\theta }{\mathbf{\theta }}^{\top }\mathit{x}{\mathit{c}}^{\top }\right]\right)\hfill \end{array}$$

Therefore,

$$\begin{array}{cc}\hfill log\overleftarrow{\nu }\left(\mathbf{\theta }\right)& =-\frac{1}{2}\mathrm{tr}\left(m_{\mathit{W}}{\mathbb{E}}_{q(\mathit{x},\mathit{y})}\left[-\mathit{S}\mathit{x}{\mathit{y}}^{\top }-\mathit{c}{\mathit{x}}^{\top }\mathbf{\theta }{\mathit{y}}^{\top }-\mathit{y}{\mathit{x}}^{\top }{\mathit{S}}^{\top }-\mathit{y}{\mathbf{\theta }}^{\top }\mathit{x}{\mathit{c}}^{\top }\right]\right)\hfill \\ & -\frac{1}{2}\mathrm{tr}\left(m_{\mathit{W}}{\mathbb{E}}_{q(\mathit{x},\mathit{y})}\left[\mathit{S}\mathit{x}{\mathit{x}}^{\top }{\mathit{S}}^{\top }+\mathit{c}{\mathit{x}}^{\top }\mathbf{\theta }{\mathbf{\theta }}^{\top }\mathit{x}{\mathit{c}}^{\top }\right]\right)+const\hfill \end{array}$$

We move terms which do not depend on $\mathbf{\theta }$ to the $const$, hence

$$\begin{array}{cc}\hfill log\overleftarrow{\nu }\left(\mathbf{\theta }\right)& =-\frac{1}{2}\mathrm{tr}\left(m_{\mathit{W}}{\mathbb{E}}_{q(\mathit{x},\mathit{y})}\left[-\mathit{c}{\mathit{x}}^{\top }\mathbf{\theta }{\mathit{y}}^{\top }-\mathit{y}{\mathbf{\theta }}^{\top }\mathit{x}{\mathit{c}}^{\top }+\mathit{c}{\mathit{x}}^{\top }\mathbf{\theta }{\mathbf{\theta }}^{\top }\mathit{x}{\mathit{c}}^{\top }\right]\right)+const\hfill \\ \hfill & =-\frac{1}{2}\mathrm{tr}\left(m_{\mathit{W}}\left[-\mathit{c}{\mathbf{\theta }}^{\top }(V_{\mathit{x},{\mathit{y}}^{\top }}+m_{\mathit{x}}{m}_{\mathit{y}}^{\top })-(V_{\mathit{x},{\mathit{y}}^{\top }}+m_{\mathit{x}}{m}_{\mathit{y}}^{\top })\mathbf{\theta }{\mathit{c}}^{\top }\right]\right)\hfill \\ \hfill & -\frac{1}{2}\mathrm{tr}\left(m_{\mathit{W}}\left[\mathit{c}\left(\mathrm{tr}\left(\mathbf{\theta }{\mathbf{\theta }}^{\top }{V}_{\mathit{x}}\right)+m_{\mathit{x}}^{\top }\mathbf{\theta }{\mathbf{\theta }}^{\top }{m}_{\mathit{x}}\right){\mathit{c}}^{\top }\right]\right)+const\hfill \\ \hfill & =-\frac{1}{2}\left[-\underset{{\mathit{z}}^{\top }}{\underbrace{{\mathit{c}}^{\top }{m}_{\mathit{W}}(V_{\mathit{x},\mathit{y}}+m_{\mathit{y}}{m}_{\mathit{x}}^{\top })}}\mathbf{\theta }-{\mathbf{\theta }}^{\top }\underset{\mathit{z}}{\underbrace{(V_{\mathit{x},\mathit{y}}+m_{\mathit{x}}{m}_{\mathit{y}}^{\top })m_{\mathit{W}}\mathit{c}}}\right]\hfill \\ \hfill & -\frac{1}{2}\left[{\theta }^{\top }\underset{D}{\underbrace{m_{\gamma }(V_{\mathit{x}}+m_{\mathit{x}}{m}_{\mathit{x}}^{\top })}}\theta \right]+const\hfill \\ \hfill & =-\frac{1}{2}\left[{\mathbf{\theta }}^{\top }D\mathbf{\theta }-{\mathit{z}}^{\top }\mathbf{\theta }-{\mathbf{\theta }}^{\top }\mathit{z}\right]+const\hfill \end{array}$$

Hence,

$$\overleftarrow{\nu }\left(\mathbf{\theta }\right)\propto \mathcal{N}({\mathsf{\Lambda }}^{-1}\mathit{z},{\mathsf{\Lambda }}^{-1})$$

where

$$\begin{array}{cc}\hfill & \mathsf{\Lambda }=m_{\gamma }(V_{\mathit{x}}+m_{\mathit{x}}{m}_{\mathit{x}}^{\top })\hfill \\ \hfill & \mathit{z}=(V_{\mathit{x}\mathit{y}}+m_{\mathit{x}}{m}_{\mathit{y}}^{\top })\mathit{c}{m}_{\gamma }\hfill \end{array}$$

Appendix A.6. Update of Message to γ

$$\begin{array}{cc}\hfill & log\overleftarrow{\nu }\left(\gamma \right)={\mathbb{E}}_{q(\mathit{x},\mathit{y})q\left(\mathbf{\theta }\right)}logf(\mathit{y},\mathit{x},\mathbf{\theta },\gamma )+const\hfill \\ \hfill & ={\mathbb{E}}_{q(\mathit{x},\mathit{y})q\left(\mathbf{\theta }\right)}\left[{log\left|\mathit{W}\right|}^{\frac{1}{2}}-\frac{1}{2}\left({(\mathit{y}-\mathit{Ax})}^{\top }\mathit{W}(\mathit{y}-\mathit{Ax})\right)\right]+const\hfill \\ \hfill & ={log\left|\mathit{W}\right|}^{\frac{1}{2}}-\frac{1}{2}\mathrm{tr}\left(\mathit{W}{\mathbb{E}}_{q(\mathit{x},\mathit{y})q\left(\mathbf{\theta }\right)}\left[\mathit{y}{\mathit{y}}^{\top }-\mathit{A}\mathit{x}{\mathit{y}}^{\top }+\mathit{A}\mathit{x}{\mathit{x}}^{\top }{\mathit{A}}^{\top }-\mathit{y}{\mathit{x}}^{\top }{\mathit{A}}^{\top }\right]\right)\hfill \\ \hfill & +const\hfill \end{array}$$

First of all, let us work out the term ${log\left|\mathit{W}\right|}^{\frac{1}{2}}$

$$\begin{array}{cc}\hfill {log\left|\mathit{W}\right|}^{\frac{1}{2}}& =\frac{1}{2}log\left|\begin{array}{ccccc}\gamma & 0& 0& \cdots & 0\\ 0& \frac{1}{ϵ}& 0& \cdots & ⋮\\ 0& 0& \frac{1}{ϵ}& \cdots & ⋮\\ ⋮& ⋮& \ddots & \ddots & ⋮\end{array}\right|\hfill \\ \hfill & =\frac{1}{2}log\gamma +\frac{1}{2}(1-M)log\left(ϵ\right)=log{\gamma }^{\frac{1}{2}}+const\hfill \end{array}$$

We split the expression under the expectation into four terms:

I:

$\mathit{W}{\mathbb{E}}_{q(\mathit{x},\mathit{y})q\left(\mathbf{\theta }\right)}\left[\mathit{y}{\mathit{y}}^{\top }\right]$

II:

$\mathit{W}{\mathbb{E}}_{q(\mathit{x},\mathit{y})q\left(\mathbf{\theta }\right)}\left[\mathit{A}\mathit{x}{\mathit{y}}^{\top }\right]$

III:

$\mathit{W}{\mathbb{E}}_{q(\mathit{x},\mathit{y})q\left(\mathbf{\theta }\right)}\left[\mathit{y}{\mathit{x}}^{\top }{\mathit{A}}^{\top }\right]$ and

IV:

$\mathit{W}{\mathbb{E}}_{q(\mathit{x},\mathit{y})q\left(\mathbf{\theta }\right)}\left[\mathit{A}\mathit{x}{\mathit{x}}^{\top }{\mathit{A}}^{\top }\right]$

Term I:

$$\begin{array}{c}\hfill \mathit{W}{\mathbb{E}}_{q(\mathit{x},\mathit{y})q\left(\mathbf{\theta }\right)}\left[\mathit{y}{\mathit{y}}^{\top }\right]=\mathit{W}\left(V_{\mathit{y}}+m_{\mathit{y}}{m}_{\mathit{y}}^{\top }\right)\end{array}$$

Recalling Equation (A6), term II:

$$\begin{array}{cc}\hfill \mathit{W}{\mathbb{E}}_{q(\mathit{x},\mathit{y})q\left(\mathbf{\theta }\right)}\left[\mathit{A}\mathit{x}{\mathit{y}}^{\top }\right]& =\mathit{W}{\mathbb{E}}_{q(\mathit{x},\mathit{y})q\left(\mathbf{\theta }\right)}\left((\mathit{S}+\mathit{c}{\mathbf{\theta }}^{\top })\mathit{x}{\mathit{y}}^{\top }\right)\hfill \\ & =\mathit{W}\left(m_{\mathit{A}}(V_{\mathit{x}{\mathit{y}}^{\top }}+m_{\mathit{x}}{m}_{\mathit{y}}^{\top })\right)\hfill \end{array}$$

Term III:

$$\begin{array}{c}\hfill \mathit{W}{\mathbb{E}}_{q(\mathit{x},\mathit{y})q\left(\mathbf{\theta }\right)}\left[\mathit{y}{\mathit{x}}^{\top }{\mathit{A}}^{\top }\right]=\mathit{W}\left((V_{\mathit{y}{\mathit{x}}^{\top }}+m_{\mathit{y}}{m}_{\mathit{x}}^{\top })m_{\mathit{A}}^{\top }\right)\end{array}$$

Term IV:

$$\begin{array}{cc}\hfill & \mathit{W}{\mathbb{E}}_{q(\mathit{x},\mathit{y})q\left(\mathbf{\theta }\right)}\left[\mathit{A}\mathit{x}{\mathit{x}}^{\top }{\mathit{A}}^{\top }\right]=\mathit{W}{\mathbb{E}}_{}\left[(\mathit{S}+\mathit{c}{\mathbf{\theta }}^{\top })\mathit{x}{\mathit{x}}^{\top }{(\mathit{S}+\mathit{c}{\mathbf{\theta }}^{\top })}^{\top }\right]\hfill \\ \hfill & =\mathit{W}{\mathbb{E}}_{q(\mathit{x},\mathit{y})q\left(\mathbf{\theta }\right)}\left[\mathit{S}\mathit{x}{\mathit{x}}^{\top }{\mathit{S}}^{\top }+\mathit{c}{\mathbf{\theta }}^{\top }\mathit{x}{\mathit{x}}^{\top }{\mathit{S}}^{\top }+\mathit{S}\mathit{x}{\mathit{x}}^{\top }\mathbf{\theta }{\mathit{c}}^{\top }+\mathit{c}{\mathbf{\theta }}^{\top }\mathit{x}{\mathit{x}}^{\top }\mathbf{\theta }{\mathit{c}}^{\top }\right]\hfill \\ \hfill & =\mathit{W}{\mathbb{E}}_{q(\mathit{x},\mathit{y})}\left[\mathit{S}\mathit{x}{\mathit{x}}^{\top }{\mathit{S}}^{\top }+\mathit{c}{m}_{\mathbf{\theta }}^{\top }\mathit{x}{\mathit{x}}^{\top }{\mathit{S}}^{\top }+\mathit{S}\mathit{x}{\mathit{x}}^{\top }{m}_{\mathbf{\theta }}{\mathit{c}}^{\top }\right]\hfill \\ \hfill & +\mathit{W}{\mathbb{E}}_{q(\mathit{x},\mathit{y})}\left[\mathit{c}({\mathit{x}}^{\top }{V}_{\theta }\mathit{x}+m_{\mathbf{\theta }}^{\top }\mathit{x}{\mathit{x}}^{\top }{m}_{\mathbf{\theta }}){\mathit{c}}^{\top }\right]\hfill \\ \hfill & =\mathit{W}{\mathbb{E}}_{q(\mathit{x},\mathit{y})}\left[m_{\mathit{A}}\mathit{x}{\mathit{x}}^{\top }{m}_{\mathit{A}}^{\top }+\mathit{c}{\mathit{x}}^{\top }{V}_{\theta }\mathit{x}{\mathit{c}}^{\top }\right]\hfill \\ \hfill & =\mathit{W}\left[m_{\mathit{A}}(V_{\mathit{x}}+m_{\mathit{x}}{m}_{\mathit{x}}^{\top })m_{\mathit{A}}^{\top }+\mathit{c}(\mathrm{tr}\left(V_{\theta }{V}_{\mathit{x}}\right)+m_{\mathit{x}}^{\top }{V}_{\theta }{m}_{\mathit{x}}){\mathit{c}}^{\top }\right]\hfill \end{array}$$

As the resulting message should depend solely on $\gamma$ we need to get rid of all terms which incorporate matrix $\mathit{W}$. We notice that

$$\begin{array}{c}\hfill \mathrm{tr}\left(\mathit{W}\mathit{M}\right)=\mathrm{tr}\left(\mathit{M}·\left(\begin{array}{ccccc}\gamma & 0& 0& \cdots & 0\\ 0& \frac{1}{ϵ}& 0& \cdots & ⋮\\ 0& 0& \frac{1}{ϵ}& \cdots & ⋮\\ ⋮& ⋮& \ddots & \ddots & ⋮\end{array}\right)\right)={\mathit{c}}^{\top }\gamma \mathit{M}\mathit{c}+const\end{array}$$

where $\mathit{M}$ is an arbitrary matrix of the same dimensionality as the matrix $\mathit{W}$ ($\mathit{M}$ denote the first element of the matrix). In this way

$$\begin{array}{cc}\hfill log\overleftarrow{\nu }\left(\gamma \right)& =log{\gamma }^{\frac{1}{2}}-\frac{\gamma }{2}{\mathit{c}}^{\top }\left[V_{\mathit{y}}+m_{\mathit{y}}{m}_{\mathit{y}}^{\top }-2m_{\mathit{A}}(V_{\mathit{x}{\mathit{y}}^{\top }}+m_{\mathit{x}}{m}_{\mathit{y}}^{\top })\right]\mathit{c}\hfill \\ \hfill & -\frac{\gamma }{2}{\mathit{c}}^{\top }\left[m_{\mathit{A}}(V_{\mathit{x}}+m_{\mathit{x}}{m}_{\mathit{x}}^{\top })m_{\mathit{A}}^{\top }+\mathrm{tr}\left(V_{\theta }{V}_{\mathit{x}}\right)+m_{\mathit{x}}^{\top }{V}_{\theta }{m}_{\mathit{x}})\right]\mathit{c}\hfill \end{array}$$

After exponentiating $log\overleftarrow{\nu }\left(\gamma \right)$ it yields the gamma distribution:

$$\overleftarrow{\nu }\left(\gamma \right)\propto {\gamma }^{\frac{1}{2}}exp\left\{-\frac{\gamma }{2}b\right\}$$

or

$$\overleftarrow{\nu }\left(\gamma \right)\propto \mathsf{\Gamma }\left(\frac{3}{2},\frac{b}{2}\right)$$

where

$$\begin{array}{cc}\hfill b& =\left(V_{\mathit{y}}+m_{\mathit{y}}{m}_{\mathit{y}}^{\top }\right)-2\left(m_{\mathit{A}}(V_{\mathit{x}{\mathit{y}}^{\top }}+m_{\mathit{x}}{m}_{\mathit{y}}^{\top })\right)\hfill \\ \hfill & +\left(m_{\mathit{A}}(V_{\mathit{x}}+m_{\mathit{x}}{m}_{\mathit{x}}^{\top })m_{\mathit{A}}^{\top }\right)+\mathrm{tr}\left(V_{\theta }\left(V_{\mathit{x}}+m_{\mathit{x}}{m}_{\mathit{x}}^{\top }\right)\right)\hfill \end{array}$$

Appendix A.7. Derivation of q(x,y)

The joint recognition distribution is given by

$$\begin{array}{cc}\hfill q(\mathit{x},\mathit{y})& \propto \overrightarrow{\nu }\left(\mathit{x}\right)\tilde{f}(\mathit{x},\mathit{y})\overleftarrow{\nu }\left(\mathit{y}\right)\hfill \\ \hfill & =\mathcal{N}\left(\mathit{x}|m_{\mathit{x}},V_{\mathit{x}}\right)\mathcal{N}\left(\mathit{y}\right|m_{\mathit{A}}\mathit{x},m_{\mathit{W}}^{-1})\mathcal{N}\left(\mathit{x}\right|\mathbf{0},{\left(m_{\gamma }{V}_{\theta }\right)}^{-1})\mathcal{N}\left(\mathit{y}|m_{\mathit{y}},V_{\mathit{y}}\right)\hfill \\ \hfill & =\mathcal{N}\left(\mathit{x}|{\mathsf{\Lambda }}^{-1}\mathit{z},{\mathsf{\Lambda }}^{-1}\right)\mathcal{N}\left(\mathit{y}|m_{\mathit{y}},V_{\mathit{y}}\right)\mathcal{N}\left(\mathit{y}\right|m_{\mathit{A}}\mathit{x},m_{\mathit{W}}^{-1})\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill \mathsf{\Lambda }& =V_{\mathit{x}}^{-1}+m_{\gamma }{V}_{\theta }\hfill \\ \hfill \mathit{z}& =V_{\mathit{x}}^{-1}{m}_{\mathit{x}}\hfill \end{array}$$

$$q(\mathit{x},\mathit{y})\propto \mathcal{N}\left(\left[\begin{array}{c}\mathit{y}\\ \mathit{x}\end{array}\right]|\left[\begin{array}{c}{m}_{\mathit{y}}\\ {\mathsf{\Lambda }}^{-1}\mathit{z}\end{array}\right],{\left[\begin{array}{cc}{V}_{\mathit{y}}^{-1}& \mathbf{0}\\ \mathbf{0}& \mathsf{\Lambda }\end{array}\right]}^{-1}\right)\mathcal{N}\left(\mathit{y}\right|m_{\mathit{A}}\mathit{x},m_{\mathit{W}}^{-1})$$

Let us rearrange the terms in the Gaussian $\mathcal{N}\left(\mathit{y}\right|m_{\mathit{A}}\mathit{x},m_{\mathit{W}}^{-1})$

$$\begin{array}{cc}\hfill & \mathcal{N}\left(\mathit{y}\right|m_{\mathit{A}}\mathit{x},m_{\mathit{W}}^{-1})\propto exp\left(-\frac{1}{2}{(\mathit{y}-m_{\mathit{A}}\mathit{x})}^{\top }{m}_{\mathit{W}}(\mathit{y}-m_{\mathit{A}}\mathit{x})\right)\hfill \\ \hfill & \propto exp\left(-\frac{1}{2}\left[{\mathit{y}}^{\top }{m}_{\mathit{W}}\mathit{y}-{\mathit{y}}^{\top }{m}_{\mathit{W}}{m}_{\mathit{A}}\mathit{x}+{\mathit{x}}^{\top }{m}_{\mathit{A}}^{\top }{m}_{\mathit{W}}{m}_{\mathit{A}}\mathit{x}-{\mathit{x}}^{\top }{m}_{\mathit{A}}^{\top }{m}_{\mathit{W}}\mathit{y}\right]\right)\hfill \end{array}$$

$$\propto \mathcal{N}\left(\left[\begin{array}{c}\mathit{y}\\ \mathit{x}\end{array}\right]|\left[\begin{array}{c}\mathbf{0}\\ \mathbf{0}\end{array}\right],{\left[\begin{array}{cc}{m}_{\mathit{W}}& -m_{\mathit{W}}{m}_{\mathit{A}}\\ -m_{\mathit{A}}^{\top }{m}_{\mathit{W}}& m_{\mathit{A}}^{\top }{m}_{\mathit{W}}{m}_{\mathit{A}}\end{array}\right]}^{-1}\right)$$

$$q(\mathit{x},\mathit{y})\propto \mathcal{N}\left(\left[\begin{array}{c}\mathit{y}\\ \mathit{x}\end{array}\right]|\left[\begin{array}{c}{m}_{\mathit{y}}\\ {\mathsf{\Lambda }}^{-1}\mathit{z}\end{array}\right],{\left[\begin{array}{cc}{V}_{\mathit{y}}^{-1}& \mathbf{0}\\ \mathbf{0}& \mathsf{\Lambda }\end{array}\right]}^{-1}\right)$$

(A9a)

$$·\mathcal{N}\left(\left[\begin{array}{c}\mathit{y}\\ \mathit{x}\end{array}\right]|\left[\begin{array}{c}\mathbf{0}\\ \mathbf{0}\end{array}\right],{\left[\begin{array}{cc}{m}_{\mathit{W}}& -m_{\mathit{W}}{m}_{\mathit{A}}\\ -m_{\mathit{A}}^{\top }{m}_{\mathit{W}}& m_{\mathit{A}}^{\top }{m}_{\mathit{W}}{m}_{\mathit{A}}\end{array}\right]}^{-1}\right)$$

(A9b)

$$=\mathcal{N}\left(\left[\begin{array}{c}\mathit{y}\\ \mathit{x}\end{array}\right]|W_q^{-1}\left[\begin{array}{c}{V}_{\mathit{y}}^{-1}{m}_{\mathit{y}}\\ V_{\mathit{x}}^{-1}{m}_{\mathit{x}}\end{array}\right],W_q^{-1}\right)$$

(A9c)

where

$$\begin{array}{c}\hfill W_q=\left[\begin{array}{cc}{m}_{\mathit{W}}+V_{\mathit{y}}^{-1}& -m_{\mathit{W}}{m}_{\mathit{A}}\\ -m_{\mathit{A}}^{\top }{m}_{\mathit{W}}& m_{\mathit{A}}^{\top }{m}_{\mathit{W}}{m}_{\mathit{A}}+\mathsf{\Lambda }\end{array}\right]\end{array}$$

The precision matrix $W_q$, to put it mildly, is quite far from a nice shape as it contains “unpleasant” matrix $m_{\mathit{W}}$ with ${ϵ}^{-1}$ on the diagonal. Let us workout the covariance matrix $V_q=W_q^{-1}$. To do this, we recall two important matrix identities:

$$\begin{array}{c}\hfill {(A+B)}^{-1}=A^{-1}-A^{-1}{(B^{-1}+A^{-1})}^{-1}{A}^{-1}\end{array}$$

(A10)

and

$$\begin{array}{cc}\hfill & {\left[\begin{array}{cc}A& B\\ C& D\end{array}\right]}^{-1}\hfill \\ \hfill & =\left[\begin{array}{cc}{(A-B{D}^{-1}C)}^{-1}& -{(A-B{D}^{-1}C)}^{-1}B{D}^{-1}\\ -D^{-1}C{(A-B{D}^{-1}C)}^{-1}& D^{-1}+D^{-1}C{(A-B{D}^{-1}C)}^{-1}B{D}^{-1}\end{array}\right]\hfill \end{array}$$

Let us denote the block elements of $W_q$ as follows:

$$\begin{array}{cc}\hfill A& =m_{\mathit{W}}+V_{\mathit{y}}^{-1}B=-m_{\mathit{W}}{m}_{\mathit{A}}\hfill \\ \hfill C& =-m_{\mathit{A}}^{\top }{m}_{\mathit{W}}D\underset{D^{*}}{\underbrace{m_{\mathit{A}}^{\top }{m}_{\mathit{W}}{m}_{\mathit{A}}+V_{\mathit{x}}^{-1}+m_{\gamma }{V}_{\theta }}}\hfill \end{array}$$

In this way,

$$\begin{array}{cc}\hfill & {(A-B{D}^{-1}C)}^{-1}={(\underset{A}{\underbrace{m_{\mathit{W}}+V_{\mathit{y}}^{-1}}}-m_{\mathit{W}}{m}_{\mathit{A}}{D}^{-*}{m}_{\mathit{A}}^{\top }{m}_{\mathit{W}})}^{-1}\hfill \\ \hfill & D=A^{-1}-A^{-1}{(A^{-1}-{\left(m_{\mathit{W}}{m}_{\mathit{A}}{D}^{-*}{m}_{\mathit{A}}^{\top }{m}_{\mathit{W}}\right)}^{-1})}^{-1}{A}^{-1}\hfill \end{array}$$

Let us work out the auxiliary terms

$$\begin{array}{cc}\hfill A^{-1}={(m_{\mathit{W}}+V_{\mathit{y}}^{-1})}^{-1}& =V_{\mathit{y}}-V_{\mathit{y}}{(m_{\mathit{V}}+V_{\mathit{y}})}^{-1}{V}_{\mathit{y}}\hfill \\ & =\underset{E}{\underbrace{m_{\mathit{V}}-m_{\mathit{V}}{(V_{\mathit{y}}+m_{\mathit{V}})}^{-1}{m}_{\mathit{V}}}}\hfill \end{array}$$

$$\begin{array}{cc}\hfill & {\left(m_{\mathit{W}}{m}_{\mathit{A}}{D}^{-*}{m}_{\mathit{A}}^{\top }{m}_{\mathit{W}}\right)}^{-1}=m_{\mathit{V}}{m}_{\mathit{A}}^{-\top }{D}^{*}{m}_{\mathit{A}}^{-1}{m}_{\mathit{V}}\hfill \\ \hfill & =m_{\mathit{V}}{m}_{\mathit{A}}^{-\top }(m_{\mathit{A}}^{\top }{m}_{\mathit{W}}{m}_{\mathit{A}}+V_{\mathit{x}}^{-1}+m_{\gamma }{V}_{\theta })m_{\mathit{A}}^{-1}{m}_{\mathit{V}}\hfill \\ \hfill & =\underset{F}{\underbrace{m_{\mathit{V}}+m_{\mathit{V}}{m}_{\mathit{A}}^{-\top }(V_{\mathit{x}}^{-1}+m_{\gamma }{V}_{\theta })m_{\mathit{A}}^{-1}{m}_{\mathit{V}}}}\hfill \end{array}$$

Hence,

$$\begin{array}{c}\hfill {(A-B{D}^{-1}C)}^{-1}=E-E{(F+E)}^{-1}E\end{array}$$

Next, let us consider $D^{-1}$, $D^{-1}C$ and $B{D}^{-1}$:

$$\begin{array}{cc}\hfill D^{-1}& =D^{-*}={\left(m_{\mathit{A}}^{\top }{m}_{\mathit{W}}{m}_{\mathit{A}}+(V_{\mathit{x}}^{-1}+m_{\gamma }{V}_{\theta })\right)}^{-1}\hfill \\ & =m_{\mathit{A}}^{-1}{m}_{\mathit{V}}{m}_{\mathit{A}}^{-\top }\hfill \\ & -m_{\mathit{A}}^{-1}{m}_{\mathit{V}}{m}_{\mathit{A}}^{-\top }{\left[m_{\mathit{A}}^{-1}{m}_{\mathit{V}}{m}_{\mathit{A}}^{-\top }+{(V_{\mathit{x}}^{-1}+m_{\gamma }{V}_{\theta })}^{-1}\right]}^{-1}{m}_{\mathit{A}}^{-1}{m}_{\mathit{V}}{m}_{\mathit{A}}^{-\top }\hfill \end{array}$$

$$\begin{array}{cc}\hfill D^{-1}C& =D^{-1}(-m_{\mathit{A}}^{\top }{m}_{\mathit{W}})\hfill \\ & =-m_{\mathit{A}}^{-1}+m_{\mathit{A}}^{-1}{m}_{\mathit{V}}{m}_{\mathit{A}}^{-\top }{\left[m_{\mathit{A}}^{-1}{m}_{\mathit{V}}{m}_{\mathit{A}}^{-\top }+{(V_{\mathit{x}}^{-1}+m_{\gamma }{V}_{\theta })}^{-1}\right]}^{-1}{m}_{\mathit{A}}^{-1}\hfill \end{array}$$

$$\begin{array}{cc}\hfill B{D}^{-1}& =(-m_{\mathit{W}}{m}_{\mathit{A}})D^{-1}\hfill \\ & =-m_{\mathit{A}}^{-\top }+m_{\mathit{A}}^{-\top }{\left[m_{\mathit{A}}^{-1}{m}_{\mathit{V}}{m}_{\mathit{A}}^{-\top }+{(V_{\mathit{x}}^{-1}+m_{\gamma }{V}_{\theta })}^{-1}\right]}^{-1}{m}_{\mathit{A}}^{-1}{m}_{\mathit{V}}{m}_{\mathit{A}}^{-\top }\hfill \end{array}$$

Although the resulting expressions do not have a nice form, we got rid of “unpleasant” matrix $m_{\mathit{W}}$.

Appendix A.8. Free Energy Derivations

In this section we describe how to compute the variational free energy of AR node $f(\mathit{y},\mathit{x},\mathbf{\theta },\gamma )$. Note that essentially AR node implements the univariate Gaussian $f(y,\mathit{x},\mathbf{\theta },\gamma )=\mathcal{N}\left(y|{\mathbf{\theta }}^{\top }\mathit{x},{\gamma }^{-1}\right)$ (Multivariate formulation is needed for bookkeeping previous states). The free energy functional is defined as

$$\begin{array}{cc}\hfill & F\left[q\right]\triangleq U\left[q\right]-H\left[q\right]\hfill \\ \hfill & U\left[q\right]\triangleq -{\mathbb{E}}_{q(\mathit{x},y)q\left(\mathbf{\theta }\right)q\left(\gamma \right)}logf\hfill \\ \hfill & H\left[q\right]\triangleq -{\mathbb{E}}_{q(\mathit{x},y)q\left(\mathbf{\theta }\right)q\left(\gamma \right)}logq\hfill \end{array}$$

At first, let us work out the entropy term $H\left[q\right]$.

$$\begin{array}{cc}\hfill H\left[q\right]& =-{\mathbb{E}}_{q(\mathit{x},y)}logq(\mathit{x},y)-{\mathbb{E}}_{q\left(\mathbf{\theta }\right)}logq\left(\mathbf{\theta }\right)-{\mathbb{E}}_{q\left(\gamma \right)}logq\left(\gamma \right)\hfill \\ \hfill & =\frac{1}{2}\left(log|2\pi e{V}_{\mathit{x}y}|+log|2\pi e{V}_{\mathbf{\theta }}|\right)\hfill \\ \hfill & -\alpha -log\beta +log\mathsf{\Gamma }\left(\alpha \right)+(1-\alpha )\psi \left(\alpha \right)\hfill \end{array}$$

where $\psi \left(\alpha \right)$ denotes digamma function.

Now, let us consider the average energy $U\left[q\right]$

$$\begin{array}{c}\hfill -{\mathbb{E}}_{q(\mathit{x},y)q\left(\mathbf{\theta }\right)q\left(\gamma \right)}\left[log\frac{{\gamma }^{1/2}}{\sqrt{2\pi }}-\frac{\gamma }{2}{(y-{\mathbf{\theta }}^{\top }\mathit{x})}^2\right]\end{array}$$

We split the expression under the expectation into two terms:

I:

$-{\mathbb{E}}_{q\left(\gamma \right)}\left[log\frac{{\gamma }^{1/2}}{\sqrt{2\pi }}\right]$

II:

$-{\mathbb{E}}_{q(\mathit{x},\mathit{y})q\left(\mathbf{\theta }\right)q\left(\gamma \right)}\left[-\frac{\gamma }{2}{(y-{\mathbf{\theta }}^{\top }\mathit{x})}^2\right]$

Term I:

$$\begin{array}{cc}\hfill -{\mathbb{E}}_{q\left(\gamma \right)}\left[log\frac{{\gamma }^{1/2}}{\sqrt{2\pi }}\right]& =-{\mathbb{E}}_{q\left(\gamma \right)}\left[\frac{1}{2}log\gamma -\frac{1}{2}log2\pi \right]=-\frac{1}{2}\left[\psi \left(\alpha \right)-log\beta \right]+\frac{1}{2}log2\pi \hfill \end{array}$$

Term II:

$$\begin{array}{cc}\hfill & -{\mathbb{E}}_{q(\mathit{x},y)q\left(\mathbf{\theta }\right)q\left(\gamma \right)}\left[-\frac{\gamma }{2}{(y-{\mathbf{\theta }}^{\top }\mathit{x})}^2\right]=\frac{m_{\gamma }}{2}{\mathbb{E}}_{q(\mathit{x},y)q\left(\mathbf{\theta }\right)}\left[{(y-{\mathbf{\theta }}^{\top }\mathit{x})}^2\right]\hfill \\ \hfill & =\frac{m_{\gamma }}{2}{\mathbb{E}}_{q(\mathit{x},y)q\left(\mathbf{\theta }\right)}\left[y^2-2y{\mathbf{\theta }}^{\top }\mathit{x}+{\mathbf{\theta }}^{\top }\mathit{x}{\mathit{x}}^{\top }\mathbf{\theta }\right]\hfill \\ \hfill & =\frac{m_{\gamma }}{2}\left[{\sigma }_y^2+m_y^2-2\left[V_{x_t{\mathit{x}}^{\top }}+m_y{m}_{\mathit{x}}^{\top }\right]m_{\mathbf{\theta }}+\mathrm{tr}\left[(V_{\mathbf{\theta }}+m_{\mathbf{\theta }}{m}_{\mathbf{\theta }}^{\top })V_{\mathit{x}}\right]\right]\hfill \\ \hfill & +\frac{m_{\gamma }}{2}\left[m_{\mathbf{\theta }}^{\top }(V_{\mathit{x}}+m_{\mathit{x}}{m}_{\mathit{x}}^{\top })m_{\mathbf{\theta }}\right]\hfill \end{array}$$

hence

$$\begin{array}{cc}\hfill U\left[q\right]& =-\frac{1}{2}\left[\psi \left(\alpha \right)-log\beta \right]+\frac{1}{2}log2\pi +\frac{m_{\gamma }}{2}d\hfill \end{array}$$

where

$$\begin{array}{cc}\hfill d& ={\sigma }_y^2+m_y^2-2\left[V_{y{\mathit{x}}^{\top }}+m_y{m}_{\mathit{x}}^{\top }\right]m_{\mathbf{\theta }}+\mathrm{tr}\left[(V_{\mathbf{\theta }+m_{\mathbf{\theta }}{m}_{\mathbf{\theta }}^{\top })V_{\mathit{x}}}\right]+m_{\mathbf{\theta }}^{\top }(V_{\mathit{x}}+m_{\mathit{x}}{m}_{\mathit{x}}^{\top })m_{\mathbf{\theta }}\hfill \end{array}$$