Probability Forecast Combination via Entropy Regularized Wasserstein Distance

Abstract

We propose probability and density forecast combination methods that are defined using the entropy regularized Wasserstein distance. First, we provide a theoretical characterization of the combined density forecast based on the regularized Wasserstein distance under the assumption. More specifically, we show that the regularized Wasserstein barycenter between multivariate Gaussian input densities is multivariate Gaussian, and provide a simple way to compute mean and its variance–covariance matrix. Second, we show how this type of regularization can improve the predictive power of the resulting combined density. Third, we provide a method for choosing the tuning parameter that governs the strength of regularization. Lastly, we apply our proposed method to the U.S. inflation rate density forecasting, and illustrate how the entropy regularization can improve the quality of predictive density relative to its unregularized counterpart.

1. Introduction

In this paper, we study a class of density forecast combination methods based on a Wasserstein metric. In the univariate case, an equally weighted centroid defined by a Wasserstein metric corresponds to a quantile averaging or vincentized center where quantiles of forecast densities are averaged. The resulting combined density tends to be narrower than the linear opinion rule [1,2,3], which may or not be desirable, depending on the context.

We propose to use the entropy regularized Wasserstein metric to construct a combined density forecast. Like its unregularized counterpart, this combined probability/density can be defined by an optimization problem, but the optimization problem in this case includes an additional regularization term that penalizes densities with low entropy, which ensures the combined density forecast is smooth. One advantage of this approach is that the entropy regularized Wasserstein barycenter can be found in a much more computationally efficient manner than its unregularized counterpart when the input densities are multi-dimensional [4].

While computational efficiency is the most commonly cited reason for using entropy regularization, this paper demonstrates that there is an additional advantage of regularization when it comes to the density combination problem. It provides a way to tune the degree of dispersion of the combined density forecast. To the best of our knowledge, this regularized metric has not been explored in the context of the density forecasting combination problem.

As a part of our discussion, we provide a theoretical characterization of the regularized Wasserstein distance under the Gaussian assumption. More specifically, we show that the regularized Wasserstein barycenter between two multivariate Gaussian inputs is multivariate Gaussian. Our proof complements Theorem 1 of [5], which characterizes the regularized Wasserstein barycenter among an arbitrary number of univariate normal densities. In addition, our result also provides a simple recursive equation that is guaranteed to converge to the variance–covariance matrix.

We proceed as follows. Section 2 formulates a density forecast combination problem with a general metric. Several existing aggregation methods in the literature can be formulated with the choice of a specific metric within this unified framework. After discussing these existing approaches, we introduce our proposal of using the entropy regularized Wasserstein barycenter. Section 3 provides theoretical results that describe the impact of entropy regularization on the combined density under a Gaussian assumption and discusses how this helps improve the quality of the combined density prediction. Section 4 discusses how to set the strength of the entropy regularization in practice and shows that our proposed selection rule achieves a certain notion of optimality. Section 5 provides an empirical exercise that illustrates how entropy regularization improves the quality of density prediction of the U.S. inflation rate relative to the unregularized combined density forecast. Section 6 concludes the article.

2. Regularized Wasserstein Barycenter for Density Forecast Combination

This section introduces the density combination problem; see, for example [6]. We assume that agent $i\in \{1,\dots ,N\}$ at time $t\in {\mathbb{N}}_{+}$ provides a forecast of the density function $p_{it}:{\mathbb{R}}^d\to {\mathbb{R}}_{+},$ with distribution function denoted by $P_{it}:{\mathbb{R}}^d\to {\mathbb{R}}_{+},$ of the random variable $y_{t+h}$ with $h\in {\mathbb{N}}_{+}$. We are interested in aggregating information contained in the N agents’ forecasts to generate a better predictive distribution for $y_{t+h}$.

Throughout the paper, we shall focus on density combinations that can be viewed as a type of average over probability densities. Specifically, those that can be defined as

$${\overline{p}}_t=arg\underset{p_t\in \mathcal{P}}{min}\sum _{i=1}^{N}D(p_{it},p_t),$$

(1)

where $D(p_i,p_j)$ is a measure of the discrepancy between the densities $p_i$ and $p_j.$ When $D(·)$ satisfies the usual properties of a distance metric, which is the case when $D(·)$ is defined as Euclidean or an unregularized Wasserstein metric, then ${\overline{p}}_t$ is known as a Fréchet mean, which is a generalization of the average for real numbers. We will refer to ${\overline{p}}_t$ as a barycenter to also encompass the more general case in which $D(·)$ is not a metric. As described in Equation (1), we restrict our attention to the case in which ${\overline{p}}_t$ is a density forecast with each input density having equal weight, which is known to perform quite well as a combination forecast [7].

A specific choice of metric, $D(p_i,p_j)$, will lead to a different combined density, ${\overline{p}}_t$. Before introducing our proposed definition of $D(·),$ the entropy regularized Wasserstein metric, the next two sections introduce choices for $D(p_i,p_j)$ that lead to well-known density forecast combination methods.

2.1. Equal-Weighted Linear Opinion Rule

As a starting point let us consider $D(p_i,p_j):={\Vert p_i-p_j\Vert }_2^2$. Then, Equation (1) becomes

$${\overline{p}}_t=arg\underset{p_t\in \mathcal{P}}{min}\sum _{i=1}^N\int {(p_{it}-p_t)}^2,$$

(2)

which results in the following solution

$${\overline{p}}_t=\frac{1}{N}\sum _{i=1}^N{p}_{it}.$$

(3)

This can be derived using the first-order condition with respect to $p_t,$ which is ${\sum }_{i=1}^N(p_{it}-{\overline{p}}_t)=0$.

This solution is known as the linear opinion rule with equal-weighting. This is the prototypical aggregation method both in the forecasting literature and in practice; see, for example [1]. This is a particularly tractable density combination method, as it is equivalent to a mixture density, and it has the additional advantage of being computationally tractable to compute. However, one disadvantage is that it does not preserve the shape of the individual forecast densities. For example, when combining two uni-modal densities, the resulting solution is generally bi-modal.

2.2. Quantile Aggregation and the Wasserstein Barycenter

In this section we consider the case in which $D(·)$ is defined as the p-Wasserstein metric, which is defined as

$$W_p(p_{it},p_{jt})={\left(\underset{\phi \in \Omega (p_{it},p_{jt})}{inf}\int \Vert z_i-z_j{\Vert }^p\mathrm{d}\phi (z_i,z_j)\right)}^{1/p},$$

(4)

where $\Omega (p_{it},p_{jt})$ is the set of all joint distributions $\phi (z_i,z_j)$ that have marginal densities given by $p_{it}$ and $p_{jt}$, respectively. Formally, we write

$$\Omega (p_{it},p_{jt})=\left\{\phi :{\mathbb{R}}^d\times {\mathbb{R}}^d\to {\mathbb{R}}_{+}^1|\forall A\subset {\mathbb{R}}^d,\phi (A,{\mathbb{R}}^d)=p_{it}\left(A\right)\mathrm{and}\phi ({\mathbb{R}}^d,A)=p_{jt}\left(A\right)\right\}.$$

(5)

In other words, each $\phi \in \Omega (p_{it},p_{jt})$ is a coupling between the distributions $p_{it}$ and $p_{jt}.$ In the optimal transport literature, the minimizer of (4) is also known as the optimal transport plan. This is because, for any $A,B\subset {\mathbb{R}}^d,$ $\phi (A,B)$ can be interpreted as the amount of mass that is moved from A to B in order to minimize $E\left({\Vert z_i-z_j\Vert }_p^p\right)$ where $z_i\sim p_{it}$ and $z_j\sim p_{jt}.$ For more detail on the field of optimal transport, see [8,9].

A special case of this Wasserstein barycenter has a close relation to a recently proposed probability/density forecast combination method in the forecasting literature. More specifically, suppose that input densities are univariate, and ${\overline{p}}_t$ is defined as the squared Wasserstein metric, denoted by $D(·):=W_2^2(·);$ in this case, we have

$${\overline{P}}_t^{-1}\left(\tau \right)=\frac{1}{N}\sum _{i=1}^N{P}_{it}^{-1}\left(\tau \right),\mathrm{for}\mathrm{all}\tau \in (0,1),$$

(6)

where $P_{it}^{-1}(·)$ and ${\overline{P}}_t^{-1}(·)$ are the quantile function of agent i and of the combination method, respectively. This forecast aggregation rule is also known as “quantile aggregation” or “Vincentized distribution” [2,3,10]. We prefer the representation of Equation (1) because this definition can be easily extended to higher dimensional densities or mixed data types (e.g., when some inputs are continuous and others are discrete) unlike quantile aggregation.

The Wasserstein barycenter is known to preserve the shape of input densities, such as log-concavity [11]. For example [12] show that the Wasserstein barycenter of the inputs, $N({\mu }_1,S_1)$ and $N({\mu }_2,S_2),$ is $N(({\mu }_1+{\mu }_2)/2,S),$ where S is the solution of,

$$S={\left(S^{1/2}{S}_1{S}^{1/2}\right)}^{1/2}/2+{\left(S^{1/2}{S}_2{S}^{1/2}\right)}^{1/2}/2;$$

(7)

see also [13]. This is different than the linear opinion rule, which leads to a mixture of two normal densities with mean $({\mu }_1+{\mu }_2)/2$ and variance $\frac{{\sigma }_1^2+{\sigma }_2^2}{2}+\frac{{({\mu }_1-{\mu }_2)}^2}{4}$, which, in contrast, can be expected to be bi-modal whenever ${\mu }_1\ne {\mu }_2$.

Another difference between these two aggregation methods is that the variance of the Wasserstein barycenter is smaller than that of the combined density resulting from a linear opinion rule. This holds for a more general class of input densities as shown in [2] in the univariate case. Of course, a narrow (i.e., sharp) predictive density can be good or bad depending on the underlying distribution of the target variable. It may be desirable to have an ability to flexibly adjust the dispersion of the combined density.

2.3. Regularized Wasserstein Barycenter

Now, we turn to our proposal. In this paper, we use a regularized Wasserstein distance [14,15] to combine individual probability forecasts. The regularization term used in this approximation of the Wasserstein metric is given by the negative differential entropy, which, when $\phi$ is an absolutely continuous measure, we will define as, $h\left(\phi \right)={\int }_{{\mathbb{R}}^d\times {\mathbb{R}}^d}log\left(\frac{\mathrm{d}\phi }{\mathrm{d}\lambda }\right)\mathrm{d}\phi ,$ where $\lambda$ is the Lebesgue measure, and infinity otherwise. We will use $h\left(\phi \right)$ to define the regularized Wasserstein metric as

$$W_{p,\gamma }(p_{it},p_{jt})={\left(\underset{\phi \in \Omega (p_{it},p_{jt})}{inf}\int \Vert z_i-z_j{\Vert }^p\mathrm{d}\phi (z_i,z_j)+\gamma h\left(\phi \right)\right)}^{1/p},$$

(8)

where $\gamma >0$ controls a strength of regularization. Note that $\phi$ is constrained by the same two marginal restrictions as its unregularized counterpart, as described in the definition of $\Omega (p_{it},p_{jt})$. This form of regularization is originally introduced by [14] in order to estimate the Wasserstein metric in a computationally efficient manner using the iterative proportional fitting procedure (IPFP) provided by [16].

When $\gamma =0$, there is no regularization, so we have $W_{p,0}(p_{it},p_{jt})=W_p(p_{it},p_{jt}).$ One can also show that the optimal coupling, say ${\phi }_{\gamma }^{🟉},$ satisfies ${lim}_{\gamma \to 0^{+}}{\phi }_{\gamma }^{🟉}={\phi }_0^{🟉}$ when ${\phi }_0^{🟉}$ is uniquely defined, and otherwise this limiting value is given by the element of the set of optimal unregularized couplings with maximum entropy [15]. Higher values of $\gamma$ place more weight on the second term in the objective function, which results in optimal couplings that are smoother and more dispersed than their unregularized counterparts.

Defining $D(p_{it},p_{jt})$ by $W_{2,\gamma }^2(p_{it},p_{jt})$ results in the combined density

$${\overline{p}}_t=arg\underset{p_t\in \mathcal{P}}{min}\sum _{i=1}^N{W}_{2,\gamma }^2(p_{it},p_{jt}),$$

(9)

which is known as the regularized Wasserstein barycenter. The authors of [4] provided a generalization of the IPFP procedure to find this barycenter that is more computationally efficient than the unregularized case. While computational efficiency is the commonly cited reason for using entropy regularization, as we will see in the later sections, our motivation for regularization is not entirely computational.

For the rest of the paper, we study this regularized Wasserstein barycenter, which is ${\overline{p}}_t$ defined in Equation (1) using (8). First, we present analytical results under a parametric assumption that broadens our understanding about the role of the regularization in forecast density combination. Then, we discuss how one can empirically choose the strength of the regularization that would achieve a certain notion of optimality.

3. Analytical Results: The Impact of Entropy Regularization

In this section we provide analytical results that describe the impact of entropy regularization on the shape of the barycenter. To better compare this barycenter with its unregularized counterpart in the Gaussian case, as defined above, we will focus on the regularized barycenter when $p_1$ and $p_2$ are d-dimensional multivariate Gaussians ($d\ge 1$). The regularized Wasserstein barycenter in this case is defined as

$$\overline{p}\in arg\underset{q}{min}\left(W_{\gamma }^2(p_1,q)+W_{\gamma }^2(p_2,q)\right).$$

(10)

The following theorem completely characterizes the resulting barycenter in this case. Like the unregularized case, the theorem shows that regularization does not impact the mean of the barycenter; however, it does have an impact on its variance–covariance matrix.

Theorem 1.

Let $p_1$ and $p_2$ be Gaussian density functions with means ${\mu }_1,{\mu }_2\in {\mathbb{R}}^d,$ and variance matrices, $S_1,S_2\in {\mathbb{R}}^{d\times d}.$ The regularized Wasserstein barycenter between $p_1$ and $p_2$ is given by the density function of $N({\mu }_B,S_B),$ where ${\mu }_B\in {\mathbb{R}}^d$ and $S_B\in {\mathbb{R}}^{d\times d}$ are defined by,

$$\begin{array}{cc}\hfill {\mu }_B:=& ({\mu }_1+{\mu }_2)/2\hfill \\ \hfill S_B:=& {\left(V/\gamma +I\right)}^{-1}\left(V/2+I\gamma /2+S_2\right){\left(V/\gamma +I\right)}^{-1}\hfill \\ \hfill =& {\left(-V/\gamma +I\right)}^{-1}\left(-V/2+I\gamma /2+S_1\right){\left(-V/\gamma +I\right)}^{-1},\hfill \end{array}$$

where $V\in {\mathbb{R}}^{d\times d}$ is the unique symmetric matrix that satisfies these equalities and $-I\gamma <V<I\gamma .$

Also, the iterates of the following series converge to V when $V^{\left(0\right)}:={\mathbf{0}}_{d\times d},$

$$V^{(k+1)}=S_2-S_1+S_1{\left(S_1+I\gamma /2-V^{\left(k\right)}/2\right)}^{-1}{S}_1-S_2{\left(S_2+I\gamma /2+V^{\left(k\right)}/2\right)}^{-1}{S}_2.$$

The proof of this result is included in the Appendix A. We prove a slightly more general version of the theorem where the objective function in Equation (10) is a weighted average of $W_{\gamma }^2(p_1,q)$ and $W_{\gamma }^2(p_2,q)$. The proof first derives a system of equations that characterizes the barycenter in the case in which the regularized barycenter is Gaussian. Afterward, a fixed point theorem provided by [17] for mappings on partially ordered sets is used to show that this system has a unique solution, and this, along with convexity of Equation (10), implies the regularized barycenter is Gaussian.

Now, we discuss our theoretical results and their implication to the density forecast combination problem.

Remark 1.

(on location). Regularization does not affect the mean of the resulting barycenter, which is a property that may not hold in the more general setting that does not include a normality assumption. For example, suppose the domain of p is $[0,1],$ and $E_{x\sim p}\left(x\right)\ne 1/2,$ and consider the barycenter between p and itself. For any fixed density function $q,$ the optimal coupling of the optimization problem that defines $W_{\gamma }^2(p,q)$ converges to $d\phi (z_1,z_2)/d\lambda =q\left(z_1\right)p\left(z_2\right),$ as this is the coupling with maximum entropy that has marginals given by q and $p;$ see for example, [15]. However, the negative entropy of $d\phi (z_1,z_2)/d\lambda =p\left(z_2\right),$ is less than or equal to that of $d\phi (z_1,z_2)/d\lambda =q\left(z_1\right)p\left(z_2\right),$ for any such fixed density $q.$ We can also ensure these couplings are feasible by defining q to be a uniform density function, so we have ${lim}_{\gamma \to \infty }q=1.$ This implies that ${lim}_{\gamma \to \infty }{E}_{x\sim q}\left(x\right)=1/2,$ regardless of the $E_{x\sim p}\left(x\right).$ Since the unregularized density is given by $q=p,$ and $E_{x\sim p}\left(x\right)\ne 1/2,$ the regularization parameter does impact the mean of the barycenter.

Remark 2.

(on dispersion) Regularization tends to smooth the resulting barycenter, leading to a more dispersed combined density. To understand this point, let us consider a simple example below.

Example 1.

Consider a case with univariate $p_{it}=N({\mu }_{it},{\sigma }^2)$ and $N=2$. Then, the original Wasserstein barycenter (quantile averaging) is ${\overline{p}}_t=N(({\mu }_{1t}+{\mu }_{2t})/2,{\sigma }^2)$. On the other hand the regularized Wasserstein barycenter is ${\overline{p}}_t\left(\gamma \right)=N(({\mu }_{1t}+{\mu }_{2t})/2,{\sigma }^2+\gamma /2).$

As this case exemplifies the strength of the regularization controls a dispersion of the combined density. The heavier the regularization the greater dispersed (or, the smoother) density we obtain. This result highlights that the entropy regularization offers an extra flexibility to control the dispersion of the combined density. In the next section, we propose a data-driven way to select the value of $\gamma$, the strength of the regularization.

Remark 3.

The normality assumption that we made to obtain the closed-form solution for the barycenter is not needed in practice. The regularized barycenter of probability/density forecasts is well-defined and computationally tractable for a broader context. One can have multiple inputs, non-Gaussian densities, discrete/continuous/mixed distribution. This includes many interesting and empirically relevant situations in economic forecasting such as macroeconomic and financial forecasting. The efficient computation of the regularized Wasserstein distance and barycenter with non-Gaussian input densities is still an active area of research. There is a large literature on computing the regularized barycenter in practice; see for example [4,18,19,20,21,22,23].

Remark 4.

During the review process for this paper, we became aware of a similar result that was proved independently of ours by [5]. There are two primary differences between these results. First, our result provides the regularized barycenter between two multivariate normal densities, while Theorem 1 in Janati et al. (2020a) provides the barycenter between an arbitrary number of univariate normal densities. Second, our result also provides a recursive formula to compute the variance–covariance matrix of the barycenter, which guarantees a convergence to a desired solution. We appreciate one of referees who pointed out relevant papers.

There have also been a number of recent results on a few related barycenters, including those that are modified to avoid the increase in the dispersion of the barycenter caused by regularization using one of the following two techniques. First, a Kullback–Leibler divergence penalty term can be used, with a reference measure given by the product of the input densities, rather than differential entropy. Second, a technique known as debiasing can also be used. For example, the remaining results in [5], as well as the results provided by [24,25], characterize these types of regularized Wasserstein barycenters between Gaussian densities. In contrast to the barycenter we consider, which can be viewed as the original discrete entropy regularized Wasserstein barycenter in the limit as the number of bins diverges, increasing the regularization parameter of these alternative barycenters either decreases or does not change the variance of the barycenter.

4. On Choosing the Strength of the Regularization

This section discusses how to choose the strength of the penalization. Our empirical strategy is to select $\gamma$ by the value that most accurately fits the observed data. To economize our notation we restrict our discussion to the 1-step-ahead prediction (i.e., $h=1$). To do so, we regard the regularized barycenter computed at time t, ${\overline{p}}_t$, as a predictive likelihood for $y_{t+1}$. This predictive likelihood interpretation of the barycenter can be formally justified by the principal-agent framework similar to the one developed by [26]. Suppose we have collected the regularized barycenters and the realized value of the target variable from the initial period $\left(1\right)$ to present $\left(t\right)$. We write this collection as ${\mathcal{I}}_t$. Then, we can define a maximum likelihood estimator for $\gamma$ at t with ${\mathcal{I}}_t$ as

$${\widehat{\gamma }}_{1:t}^{mle}\in arg\underset{\gamma \ge 0}{max}\sum _{\tau =1}^{t-1}log{\overline{p}}_{\tau }(y_{\tau +1};\gamma ),$$

(11)

and the combined density prediction for $y_{t+1}$ at time t is

$$\widehat{p}\left(y_{t+1}\right|{\mathcal{I}}_t)={\overline{p}}_t(y_{t+1};{\widehat{\gamma }}_{1:t}^{mle}).$$

(12)

There is a notion in which this combined density with $\widehat{\gamma }$ is optimal. Suppose that $y_t{\sim }_{i.i.d.}{p}^{*}\left(y\right)$, and assume that forecasters report a sequence of predictive densities, $p_i\left(y\right)$ for $y_t$, $t=1,2,\dots ,T$ and $i=1,2,\dots ,N$. These forecasts are reported before the realization of $y_t$, and the barycenter $\overline{p}(y;\gamma )$ is defined by $p_i\left(y\right)$’s and $\gamma >0$. Then, the following can be shown under regularity conditions,

$$\frac{1}{T}\sum _{t=1}^{T}log\overline{p}(y_t;\gamma ){\to }_p\int log\overline{p}(y;\gamma )p^{*}\left(y\right)dy\mathrm{as}T\to \infty ,$$

for $\gamma \in \Gamma \in {\mathbb{R}}_{+}$. In turn, a maximizer of the left-hand-side term also converges to the maximizer of the right-hand-side term, which is a minimizer of

$$KL(\overline{p}(y;\gamma ),p^{*}\left(y\right))=-\int log\overline{p}\left(y\right)p^{*}\left(y\right)dy+\int log\left(p^{*}\left(y\right)\right)p^{*}\left(y\right)dy.$$

Therefore, $\widehat{\gamma }$ converges to the pseudo-true parameter that minimizes Kullback–Leibler (KL) divergence from the regularized barycenter to the true data generating process. In other words, we find $\gamma$ that makes the resulting barycenter close to the true data generating process in the limit. This asymptotic thought experiment can be justifiable under quite general conditions, allowing for a range of serial dependence in $y_t$ as well as a flexible form of the regularized Wasserstein barycenter implied by $p_{i,t-1}\left(y_t\right)$’s. We can operationalize this by recognizing that ${\overline{p}}_{t-1}(y;\gamma )$ can be viewed as a predictive likelihood for $y_t$ formed at time $t-1$. Then, quasi-MLE theory can be invoked, e.g., [27,28]. We provide a simple example in which the true data generating process follows the autoregressive (AR) process.

Example 2.

Suppose that forecaster 1 and 2 use mean-zero Gaussian AR(1) process to construct their density prediction. The two forecasts differ only by the mean reversion parameter. That is, the means of predictive distribution for forecaster 1 and 2 are ${\mu }_{1t}={\rho }_1{y}_{t-1}$ and ${\mu }_{2t}={\rho }_2{y}_{t-1}$, respectively. Based on our theory in the previous section, the barycenter is ${\overline{p}}_{t-1}(y;\gamma )=N({\overline{\mu }}_t,{\sigma }^2+\gamma /2)$ where ${\overline{\mu }}_t=({\mu }_{1t}+{\mu }_{2t})/2$, and the log density of the regularized barycenter at τ for $y_{\tau +1}$ is

$$log\left({\overline{p}}_{\tau }(y_{\tau +1};\gamma )\right)=-1/2log\left(2\pi \right)-1/2log({\sigma }^2+\gamma /2)-1/2{\left(\frac{y_{\tau +1}-{\overline{\mu }}_{\tau +1}}{\sqrt{{\sigma }^2+\gamma /2}}\right)}^2,$$

(13)

and the ML estimator for γ at time t is

$${\widehat{\gamma }}_{1:t}^{mle}\in arg\underset{\gamma \ge 0}{max}\sum _{\tau =1}^{t-1}\left(-1/2log\left(2\pi \right)-1/2log({\sigma }^2+\gamma /2)-1/2{\left(\frac{y_{\tau +1}-{\overline{\mu }}_{\tau +1}}{\sqrt{{\sigma }^2+\gamma /2}}\right)}^2\right),$$

(14)

which leads to

$${\widehat{\gamma }}_{1:t}^{mle}=2\times max\left(\frac{1}{(t-1)}\sum _{\tau =1}^{t-1}{(y_{\tau +1}-{\overline{\mu }}_{\tau +1})}^2-{\sigma }^2,0\right).$$

(15)

Now, suppose that the actual data generating process is

$$y_t={\rho }_{*}{y}_{t-1}+v_t,v_t{\sim }_{i.i.d.}N(0,{\sigma }_{*}^2).$$

(16)

When the simple average of both forecasters’ autoregressive parameter equals ${\rho }_{*}$, the ML estimate for γ depends on the true conditional variance, ${\sigma }_{*}^2$, and forecasters’ conditional variance. If the sample variance is larger than that of the forecasters, then γ is chosen so that the resulting regularized barycenter has the same variance as the sample variance. On the other hand, if the sample variance is smaller than that of the forecasters, then γ is set to 0. Note that there is an asymmetry in adjusting the variance of the barycenter. This is natural in that the regularization only makes the resulting density smoother. In practice, this may not be a problem if the practitioner’s concern is the combined density being too sharp (e.g., relative to the linear opinion rule).

Note that ${\widehat{\gamma }}_{1:t}^{mle}$ converges in probability to ${\gamma }_{\infty }=2max({\sigma }_{*}^2-{\sigma }^2,0)$. The KL divergence between $\overline{p}(y_{t+1};\gamma )$ and the true conditional density of $y_{t+1}$ at t is minimized at $\gamma ={\gamma }_{\infty }$. This confirms that our selection rule for γ aims to fit the data well by shaping the regularized barycenter as close as possible to the data generating process.

5. Empirical Illustration

In this section, we illustrate our proposed method using macroeconomic data for the U.S. We consider 14 hypothetical forecasters who produce their own 1-step-ahead forecast about the U.S. inflation rate based on the following vector autoregression (VAR) with three variables,

$$Y_t={\Phi }_0+\sum _{i=1}^4{\Phi }_i{Y}_{t-i}+e_t,e_t{\sim }_{i.i.d}N(0,\sum ),$$

(17)

where $Y_t$ is a $3\times 1$ vector that consists three quarterly macroeconomic variables, ${\Phi }_0$ is a $3\times 1$ vector, ${\Phi }_1,{\Phi }_2,{\Phi }_3,{\Phi }_4,\sum$ are $3\times 3$ matrices. The first two elements of $Y_t$ are common to all 14 forecasters: the annualized quarter-over-quarter inflation rate and real GDP growth rate. They differ by the third element of $Y_t$. We assign each forecaster a different macroeconomic variable from the FRED-QD database by [29]. A detailed description of the variable used in this exercise is in

Table 1. Variables used in empirical exercises.

$Y^{\left(i\right)}=[Y_1,Y_2,Y_3^{\left(i\right)}]$	Used by	Variable Description	FRED-QD Mnemonic
Variable 1 $\left(Y_1\right)$	All	Inflation rate	GDPCTPI
Variable 2 $\left(Y_2\right)$	All	Real GDP growth rate	GDPC1
Variable 3 $\left(Y_3^{\left(i\right)}\right)$	Forecaster 1	Real Personal Consumption Expenditures	PCECC96
	Forecaster 2	Industrial Production Index	INDPRO
	Forecaster 3	All Employees: Total Nonfarm	PAYEMS
	Forecaster 4	Housing Starts: Total Privately Owned Housing Units Started	HOUST
	Forecaster 5	Real Manufacturing and Trade Industries Sales	CMRMTSPLx
	Forecaster 6	Real Crude Oil Prices: West Texas Intermediate (WTI)	OILPRICEx
	Forecaster 7	Real Average Hourly Earnings: Manufacturing	CES3000000008x
	Forecaster 8	10-Year Treasury Constant Maturity Minus 3-Month Treasury Bill	GS10TB3Mx
	Forecaster 9	Real Commercial and Industrial Loans	BUSLOANSx
	Forecaster 10	Real Total Assets of Households and Nonprofit Organizations	TABSHNOx
	Forecaster 11	U.S. / U.K. Foreign Exchange Rate	EXUSUKx
	Forecaster 12	Consumer Sentiment (University of Michigan)	UMCSENTx
	Forecaster 13	S&P’s Common Stock Price Index: Composite	S&P 500
	Forecaster 14	Real Disposable Business Income	CNCFx

Note: All variables are obtained from the FRED-QD database [29]. Inflation rate is computed as a log difference of the GDP deflator (GDPCTPI). Real GDP growth rate is computed as a log difference of the real GDP (GDPC1). All other variables are transformed following [29]. We use the 2019–11 vintage data. Each forecaster constructs a predictive distribution using their own vector autoregression with three variables $Y^{\left(i\right)}=[Y_1,Y_2,Y_3^{\left(i\right)}]$ where $i=1,2,\dots ,14$.

.

We compute each forecasters’ 1-step-ahead predictive distribution for the inflation rate at time t as ${\pi }_{t+1|t}\sim N({\left[{\mu }_{t+1|t}\right]}_{(1,1)},{\left[{\sum }_{t+1|t}\right]}_{(1,1)})$ where ${\left[x\right]}_{(i,j)}$ denotes $(i,j)$ element of vector/matrix x. These forecasters assume that the 1-step-ahead predictive distribution of $Y_{t+1}$ at t is Gaussian, and they use their best guess about the predictive mean and variance to construct the predictive distribution. More specifically, they set these two moments as

$${\mu }_{t+1|t}={\widehat{\Phi }}_{0,t}+\sum _{p=1}^4{Y}_{t-p+1}^{\prime }{\widehat{\Phi }}_{p,t},\mathrm{and}{\sum }_{t+1|t}={{\displaystyle \widehat{\sum }}}_t,$$

(18)

where $({\widehat{\Phi }}_{0,t},{\widehat{\Phi }}_{1,t},{\widehat{\Phi }}_{2,t},{\widehat{\Phi }}_{3,t},{\widehat{\Phi }}_{4,t},{{\displaystyle \widehat{\sum }}}_t)$ is the posterior mean of $p({\Phi }_0,{\Phi }_1,{\Phi }_2,{\Phi }_3,{\Phi }_4,\sum |Y_{t:(t-R+1)})$ with a flat prior. We set $R=80,$ meaning that they also use the most recent 20 years of data to construct the predictive distribution.

We let the forecasters to generate their 1-step-ahead predictive distribution for the inflation rate from 2001Q1 to 2018Q4. This leaves us 72 quarters for a forecast evaluation sample. At each point in time, we also combine these 14 predictive densities based on the regularized Wasserstein barycenter with 20 different values of the regularization parameter $\gamma$ on $[0.3,10]$. As we explained in the previous section, a larger value of this parameter implies a stronger regularization, and the resulting combined predictive density becomes smoother with a larger variance. We also compute the combined density with $\gamma =0$ , which leads to “quantile aggregation” or “Vincentized distribution”. Our computation of the regularized barycenter is based on the algorithm developed and proposed by [19]. The MATLAB toolbox that implements this algorithm is available from https://github.com/gpeyre/2015-SIGGRAPH-convolutional-ot.

We evaluate each forecaster’s, and other forecast aggregation, methods by the sum of log predictive score, which is a logarithm of the predictive density evaluated at the actualized value, over the evaluation sample. These results are presented in Figure 1. The left panel presents the sum of the log score for individual forecasters sorted by their performance. There is a sizeable difference in their historical performance. The solid line represents the performance based on the quantile aggregation, which aggregates all forecasters in the pool. As found by other research papers, e.g., [2,3] the quantile aggregation method generates a decent predictive distribution, which performs slightly better than the ex-post top 4 forecaster.

The right panel in Figure 1 shows the historical performance of our proposed approach with various choices of regularization parameter, $\gamma$. For a wide range of values for $\gamma$ the regularized barycenter performs better than the quantile aggregation. It does even better than the best individual. This is interesting because we cannot identify the best forecaster a priori.

The optimal value of $\gamma$ defined in Equation (11) at the end of the evaluation sample would be the value of $\gamma$ that corresponds to the peak of the curve, which is about ${\widehat{\gamma }}_{2018Q4}\approx 1.3$. If we were to use this value at the beginning of the evaluation sample, then the mean difference in the log predictive score between the regularized Wasserstein barycenter and the quantile aggregation would have been 0.12 with the heteroscedasticity and autocorrelation consistent (HAC) standard error being 0.07. This implies that the difference in the peak of the curve and the solid line is statistically significant at 10% confidence level.

To make the $\gamma$ selection fully adaptive, we also compute the optimal $\gamma$ sequentially from the beginning to the end of the evaluation sample. That is, we set the predictive density for $y_{t+1}$ as the regularized barycenter with the value of $\gamma$ that maximizes the objective function defined in Equation (11) only using the information available from the beginning of the sample up to t. In this way, we do not use any future information when choosing the value of $\gamma$. Even in this case the regularized Wasserstein barycenter performs better than the best individual forecaster and the quantile aggregation. The sum of the log predictive score is −93.09, and the mean difference in the log predictive score with the quantile aggregation is 0.11 with the HAC standard error being 0.06. This suggests that the regularized Wasserstein barycenter with the adaptively chosen (e.g., estimated online) $\gamma$ performs statistically better than its unregularized counterpart, the quantile aggregation, at the 10% significance level. This superior predictive performance of the regularized Wasserstein barycenter relative to the quantile aggregation remains unchanged even when we split the evaluation sample into two. The mean difference in the log predictive score is 0.13 and 0.09 for the first half and the second half of the evaluation sample, respectively.

6. Concluding Remarks

This paper proposes to use the entropy regularized Wasserstein barycenter to combine several probability and density forecasts. The entropy regularization smooths the resulting combined forecast, and it offers a flexible way to adjust the dispersion of the predictive density when it is needed. We study the effect of the regularization on the combined density forecast and provide an exact relationship between the strength of the regularization and the variance–covariance matrix of the combined density when input densities are Gaussian. We then provide a way to select the strength of regularization by choosing the regularized barycenter that most closely matches the data. We apply our proposed methodology to the U.S. inflation density forecasting and show how the entropy regularization can improve the quality of the density forecast relative to its unregularized counterpart.

In this article, we restrict weights of each input densities on the final combined density to be pre-determined at some values (i.e., equal weighting). This choice was intentional to focus on studying the role of entropy regularization. In practice, however, it is possible that a subset of input densities might be superior to others, and one may wish to put different weights on each input density. Alternatively, it is desirable to include only a subset of input densities into the combined density and set other weights to zero, see, for example, [30]. For those cases, it is fruitful to develop a data-dependent method that chooses both the regularization strength and those weights simultaneously, which is a topic for future research.