Decision-Making Under Model Misspecification: DRO with Robust Bayesian Ambiguity Sets

Abstract

Distributionally Robust Optimisation (DRO) protects risk-averse decision-makers by considering the worst-case risk within an ambiguity set of distributions based on the empirical distribution or a model. To further guard against finite, noisy data, model-based approaches admit Bayesian formulations that propagate uncertainty from the posterior to the decision-making problem. However, when the model is misspecified, the decision-maker must stretch the ambiguity set to contain the data-generating process (DGP), leading to overly conservative decisions. We address this challenge by introducing DRO with Robust ayesian Ambiguity Sets (DRO-RoBAS) to model misspecification. These are Maximum Mean Discrepancy ambiguity sets centred at a robust posterior predictive distribution that incorporates beliefs about the DGP. We show that the resulting optimisation problem obtains a dual formulation in the Reproducing Kernel Hilbert Space and we give probabilistic guarantees on the tolerance level of the ambiguity set. Our method outperforms other Bayesian and empirical DRO approaches in out-of-sample performance on the Newsvendor and Portfolio problems with various cases of model misspecification.

1. Introduction

Decision-makers frequently encounter the challenge of optimising under uncertainty since the data-generating process (DGP) is not fully known. As a result, they must rely on available data and model families to estimate the DGP within the optimisation objective. However, the data may be noisy, the data distribution might change with time, or the model might be misspecified or poorly fitted, leading to distributional uncertainty. Decision-making in this setting is a critical challenge in various applications such as inventory planning [1], portfolio optimisation [2] and distribution shifts in machine learning applications [3].

A risk-averse decision-maker might choose to hedge against distributional uncertainty by finding the decision that minimises the worst-case risk over a set of distributions. This worst-case protection is at the heart of Distributionally Robust Optimisation (DRO) that defines an ambiguity set of distributions with respect to an estimator of the DGP. This can be fully data-driven, using the empirical measure of the observations (see, e.g., [4,5,6,7]), or model-based when expert knowledge is available to fit a model to available data, resulting in a model-based estimator for the DGP [8,9,10]. Both approaches are sensitive to the choice of DGP estimator and additional estimation error might exist in model-based DRO due to a poor fit or model uncertainty.

To overcome this, recently developed Bayesian formulations of DRO use posterior beliefs to inform the optimisation problem [11] or the ambiguity set itself [12]. However, these methods inherit the sensitivity of Bayesian posteriors to model misspecification (see, e.g., [13,14]). A key goal in DRO methodology is to choose the size of the ambiguity set such that the DGP falls within it with high probability, as illustrated in Figure 1 (left) for the two formulations of Bayesian Ambiguity Sets (BAS) [12] and our proposed Robust BAS (RoBAS). If the estimate is not accurate—for example, when the model is misspecified—then a much larger size will be required to contain the DGP as illustrated in Figure 1 (right). The price to pay for this large size is the inclusion of many probability distributions that are unlikely to occur, and which could be very pessimistic with respect to the objective function, leading to an overly conservative decision. If the decision-maker wrongly assumes the model is well specified and incorrectly chooses an overly optimistic ambiguity set size, then the DGP may not lie in the set, and the decision could be overly optimistic compared to the out-of-sample outcome, often referred to as the optimiser’s curse [4].

Uncertainty over the DGP is extensively studied outside stochastic optimisation. Consider a parametric model ${\mathbb{P}}_{\theta }$, indexed by the parameter of interest $\theta$. In the Bayesian framework, uncertainty about the parameter is typically expressed directly through prior beliefs. However, recent work in robust Bayesian inference by Lyddon et al. [15] takes a different approach: uncertainty about the parameter is now induced by uncertainty, in the form of prior beliefs, in the DGP. This concept lies at the core of the Bayesian Nonparametric Learning (NPL) framework [15,16] which relaxes the well-specified model assumption imposed by standard Bayesian inference. In this spirit, we approach the challenge of DRO under model misspecification by extending the recently proposed DRO-BAS [12] framework to tackle model misspecification through a robust NPL posterior coupled with the Maximum Mean Discrepancy (MMD) inside the ambiguity set, thus introducing DRO with Robust Bayesian Ambiguity Sets (DRO-RoBAS). While DRO-BAS targets distributional uncertainty with respect to the DGP, the robustness offered is not sufficient under model misspecification, see Figure 1.

2. Background

Let $x\in \mathcal{X}\subseteq {\mathbb{R}}^d$ be a decision-making variable for the cost function $f:\mathcal{X}\times \mathrm{\Xi }\to \mathbb{R}$ with data space $\mathrm{\Xi }\subseteq {\mathbb{R}}^D$. Let ${\left\{{\xi }_i\right\}}_{i=1}^n\stackrel{\mathrm{iid}}{\sim }{\mathbb{P}}^{\star }$ be observations from the DGP ${\mathbb{P}}^{\star }\in \mathcal{P}\left(\mathrm{\Xi }\right)$, where $\mathcal{P}\left(\mathrm{\Xi }\right)$ denotes the space of Borel probability measures on $\mathrm{\Xi }$. Furthermore, consider a parametric model family ${\mathcal{P}}_{\mathrm{\Theta }}:=\{{\mathbb{P}}_{\theta }:\theta \in \mathrm{\Theta }\}\subset \mathcal{P}\left(\mathrm{\Xi }\right)$ indexed by parameter of interest $\theta \in \mathrm{\Theta }\subseteq {\mathbb{R}}^k$. We say the model is misspecified if ${\mathbb{P}}^{\star }\notin {\mathcal{P}}_{\mathrm{\Theta }}$. DRO methods construct an ambiguity set $\mathcal{A}$, based on an estimator of ${\mathbb{P}}^{\star }$, called the nominal distribution, and minimise the worst-case expected cost over $\mathcal{A}$.

The DRO literature typically categorises ambiguity sets into two classes [17]: moment-based and discrepancy-based. The former contain distributions that satisfy constraints on the moments of ${\mathbb{P}}^{\star }$, without necessarily considering an estimator. In contrast, discrepancy-based ambiguity sets consist of distributions close to the nominal according to a specified discrepancy measure. Examples include Integral Probability Metrics (IPMs) [18], such as the Wasserstein distance [4] and the MMD [5], as well as $\varphi$-divergences like the Kullback–Leibler (KL) divergence [7]. Regardless of the choice of ambiguity set, the resulting minimax problem can be seen as a game between the decision-maker who chooses x and an adversary who chooses the worst-case distribution in $\mathcal{A}$:

$$\begin{array}{c}\hfill \underset{x\in \mathcal{X}}{min}\underset{\mathbb{P}\in \mathcal{A}}{sup}{\mathbb{E}}_{\xi \sim \mathbb{P}}\left[f_x\left(\xi \right)\right]\end{array}$$

(1)

where $f_x\left(\xi \right):=f(x,\xi )$. Although most discrepancy-based DRO methods are fully empirical, i.e., the estimator is obtained only via ${\xi }_{1:n}$, sometimes, such as in regression settings, the decision-maker needs to model the variables’ relationship via a model family ${\mathcal{P}}_{\mathrm{\Theta }}$, describing the DGP. Model-based DRO methods (e.g., [8,9,10]) use the observations to obtain an estimator ${\mathbb{P}}_{\widehat{\theta }}\in {\mathcal{P}}_{\mathrm{\Theta }}$ and use this as the nominal distribution. Thus, a poorly chosen ${\mathbb{P}}_{\widehat{\theta }}$ far from ${\mathbb{P}}^{\star }$ (in some distance sense) requires a large $\mathcal{A}$, leading to overly pessimistic decisions. This has led to Bayesian formulations of DRO which propagate uncertainty about $\theta$ in the optimisation problem.

2.1. Bayesian Formulations of DRO

Shapiro et al. [11] introduced Bayesian DRO (BDRO) which defines an expected worst-case risk objective:

$$\begin{array}{c}\hfill \underset{x\in \mathcal{X}}{min}{\mathbb{E}}_{\mathrm{\Pi }(\theta \mid {\xi }_{1:n})}\left[\underset{{\displaystyle \mathbb{P}:d_{\mathrm{KL}}\left(\mathbb{P}\right||{\mathbb{P}}_{\theta })\le ϵ}}{sup}{\mathbb{E}}_{\xi \sim \mathbb{P}}\left[f_x\left(\xi \right)\right]\right]\end{array}$$

(2)

where $\mathrm{\Pi }(\theta \mid {\xi }_{1:n})$ denotes the parameter posterior distribution for model family ${\mathcal{P}}_{\mathrm{\Theta }}$. However, risk-averse decision-makers are interested in worst-case risk formulations. For this reason, Dellaporta et al. [12] proposed two formulations of the DRO with Bayesian Ambiguity Sets (DRO-BAS) that correspond to a worst-case optimisation problem with ambiguity sets informed by the standard Bayesian posterior. In particular, they defined DRO-BAS_PP:

$$\begin{array}{c}\hfill \underset{x\in \mathcal{X}}{min}\underset{{\displaystyle \mathbb{P}:d_{\mathrm{KL}}\left(\mathbb{P}\left|\right|{\mathbb{E}}_{\mathrm{\Pi }(\theta \mid {\xi }_{1:n})}\left[{\mathbb{P}}_{\theta }\right]\right)\le ϵ}}{sup}{\mathbb{E}}_{\xi \sim \mathbb{P}}\left[f_x\left(\xi \right)\right],\end{array}$$

(3)

based on a KL-based ambiguity set with nominal distribution the posterior predictive, and DRO-BAS_PE:

$$\begin{array}{c}\hfill \underset{x\in \mathcal{X}}{min}\underset{{\displaystyle \mathbb{P}:{\mathbb{E}}_{\mathrm{\Pi }(\theta \mid {\xi }_{1:n})}[d_{\mathrm{KL}}\left(\mathbb{P}\right||{\mathbb{P}}_{\theta }\left)\right]\le ϵ}}{sup}{\mathbb{E}}_{\xi \sim \mathbb{P}}\left[f_x\left(\xi \right)\right].\end{array}$$

(4)

which considers the expected KL under the posterior distribution. The authors showcased improved out-of-sample robustness compared to BDRO in a number of Exponential family models. Although DRO-BAS in the standard Bayesian setting offers an intuitive, posterior-informed ambiguity set, it can be severely affected by model misspecification. Indeed, BAS_PE only considers probability measures $\mathbb{P}$ that are absolutely continuous with respect to ${\mathbb{P}}_{\theta }$ (denoted by $\mathbb{P}\ll {\mathbb{P}}_{\theta }$) and also admit an expected KL divergence close enough to ${\mathbb{P}}_{\theta }$. Since the expectation is informed by the posterior, a non-robust posterior will likely lie far away from the DGP. Similarly, BAS_PP considers probability measures $\mathbb{P}$ that admit small KL-divergence with respect to ${\mathbb{P}}_n^{\mathrm{pred}}$, where ${\mathbb{P}}_n^{\mathrm{pred}}:={\mathbb{E}}_{\theta \sim \mathrm{\Pi }(\theta \mid {\xi }_{1:n})}\left[{\mathbb{P}}_{\theta }\right]$ and $\mathbb{P}\ll {\mathbb{P}}_n^{\mathrm{pred}}$. Hence, the sensitivity of the Bayesian posterior will propagate to the posterior predictive and the resulting ambiguity set. A similar observation was made by Shapiro et al. [11] for the BDRO method in the misspecified case.

To remedy this, we exploit the flexibility of the DRO-BAS framework which allows us to choose a different posterior distribution and discrepancy measure, suitable for model misspecification. The notion of targeting a different discrepancy measure, other than the KL divergence, to induce robustness in the Bayesian posterior has been well established in the Bayesian inference literature. The NPL posterior [15,16] was introduced to, among others, remedy the sensitivity of Bayesian inference to model misspecification by removing the assumption that the model is correct. This is done by setting uncertainty, via nonparametric prior beliefs, directly on the DGP rather than on the parameter of interest. Incorporating DGP uncertainty in decision-making has also been considered by Wang et al. [19], who explored a nonparametric Dirichlet Process (DP) model for the DGP. Unlike the current paper, this work is not suited for parametric models and considers a weighted objective, with only one counterpart corresponding to a worst-case risk. We focus on decision-making under parametric models, which are especially useful for interpretability in decision-making, while also incorporating nonparametric prior beliefs about the DGP.

To achieve this, we leverage the work of Dellaporta et al. [20] who extended the NPL posterior to discrepancy-based loss functions and showed robustness guarantees when the Maximum Mean Discrepancy (MMD) is used. Using an MMD-based loss allows us to also employ the MMD to construct the ambiguity set. NPL is a natural choice for the DRO-BAS framework as distributional uncertainty in DRO stems directly from uncertainty in the DGP. Before we introduce the NPL posterior and the DRO-RoBAS framework, we first give a brief overview of robust Bayesian inference methodologies based on divergencies.

2.2. Robust Bayesian Inference via Divergences

To mitigate the lack of robustness in standard Bayesian inference, a class of approaches known as Generalised Bayesian Inference (GBI) [21,22,23] has been introduced. In this framework, one replaces the log-likelihood with a general empirical loss function $l_n:{\mathrm{\Xi }}^n\times \mathrm{\Theta }\to \mathbb{R}$, together with a learning rate $\beta >0$. The resulting posterior distribution for prior $\pi \left(\theta \right)$ is defined through the density

$$\begin{array}{c}\hfill {\mathrm{\Pi }}_{\mathrm{GBI}}\left(\theta \right|{\xi }_{1:n})\propto exp\left(-\beta l_n({\xi }_{1:n},\theta )\right)\pi \left(\theta \right).\end{array}$$

In most cases, the loss function $l_n$ is chosen in relation to the model family ${\mathcal{P}}_{\mathrm{\Theta }}$. In particular, the choice of $l_n({\xi }_{1:n},\theta ):=-logp({\xi }_{1:n},\theta )$ and $\beta =1$ recovers the standard Bayesian posterior. This observation highlights that the sensitivity of the standard Bayesian posterior to model misspecification stems from this specific choice of loss. To improve robustness, alternative generalised posteriors have been proposed by selecting discrepancies D with desirable properties and select $l_n$ induced by an approximation of $D({\mathbb{P}}^{\star },{\mathbb{P}}_{\theta })$ using observed data. Examples of such works based on robust divergences or discrepancies include [24,25,26,27]. Beyond GBI approaches, Bayesian NPL provides an alternative generalised formulation of Bayesian inference, which in some cases is also grounded in discrepancy-based loss functions. Unlike GBI methods, NPL places uncertainty directly on the DGP, thereby enhancing robustness to model misspecification. As distributional robustness similarly seeks to account for uncertainty in the DGP, we adopt this framework in the present work and introduce it below.

Robust NPL Posterior

In this work, we propose an alternative formulation of DRO-BAS, based on the robust NPL posterior introduced by Lyddon et al. [15] and Fong et al. [16]. We introduce a DP prior $\mathbb{Q}\sim DP(\alpha ,\mathbb{F})$ on the DGP ${\mathbb{P}}^{\star }$ where $\alpha >0$ and $\mathbb{F}\in \mathcal{P}\left(\mathrm{\Xi }\right)$. Here, $\mathbb{F}$ represents our prior beliefs about the DGP and the concentration parameter $\alpha$ dictates the strength of the beliefs with $\alpha =0$ representing a non-informative prior. To see this, note that given data ${\xi }_{1:n}$, the posterior is

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill \mathbb{Q}|{\xi }_{1:n}& \sim DP({\alpha }^{\prime },{\mathbb{F}}^{\prime }),\hfill \\ \hfill {\alpha }^{\prime }:=\alpha +n,{\mathbb{F}}^{\prime }& :={\displaystyle \frac{\alpha }{\alpha +n}}\mathbb{F}+{\displaystyle \frac{n}{\alpha +n}}{\mathbb{P}}_n\hfill \end{array}\end{array}$$

(5)

where ${\mathbb{P}}_n:={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{\delta }_{{\xi }_i}$ is the empirical measure and ${\delta }_{\xi }$ denotes the Dirac measure at $\xi \in \mathrm{\Xi }$. For $\alpha =0$, the DP posterior is centred directly on ${\mathbb{P}}_n$. If ${\mathbb{P}}^{\star }$ was known, we could directly compute:

$$\begin{array}{c}\hfill {\theta }_L\left({\mathbb{P}}^{\star }\right):=\underset{\theta \in \mathrm{\Theta }}{arg\; min}{\mathbb{E}}_{\xi \sim {\mathbb{P}}^{\star }}\left[L(\xi ;\theta )\right]\end{array}$$

(6)

where $L:\mathrm{\Xi }\times \mathrm{\Theta }\to \mathbb{R}$ denotes any loss function. Note that this NPL objective does not assume that the model is well specified but simply looks for the most likely value of $\theta$ under the expectation of the DGP or, equivalently, the parameter value that best describes the data under the candidate model. Since ${\mathbb{P}}^{\star }$ is unknown and we instead have a nonparametric posterior over it, we can propagate our posterior beliefs to the parameter of interest through the push-forward measure ${\left({\theta }_L\right)}_{\#}(DP({\alpha }^{\prime },{\mathbb{F}}^{\prime }))$ to give a posterior ${\mathrm{\Pi }}_{\mathrm{NPL}}$ on $\mathrm{\Theta }$. Sampling from this posterior can be done through the Posterior Bootstrap [16]: For B Posterior Bootstrap iterations, at iteration $j\in \left[B\right]$:

• Sample ${\mathbb{Q}}^{\left(j\right)}$ from the posterior $DP({\alpha }^{\prime },{\mathbb{F}}^{\prime })$.
• Compute ${\theta }^{\left(j\right)}={\theta }_L\left({\mathbb{Q}}^{\left(j\right)}\right)$ where ${\theta }_L(·)$ as in (6).

Dellaporta et al. [20] suggested using a discrepancy-based loss function in (6) which we introduce below.

2.3. Maximum Mean Discrepancy

The MMD belongs to the family of IPMs [28]. Let ${\mathcal{H}}_k$ be a Reproducing Kernel Hilbert Space (RKHS), for kernel $k:\mathrm{\Xi }\times \mathrm{\Xi }\to \mathbb{R}$ and norm ${\parallel ·\parallel }_k$. For ${\mathcal{P}}_k\left(\mathrm{\Xi }\right):=\{\mathbb{P}\in \mathcal{P}\left(\mathrm{\Xi }\right):{\int }_{\mathrm{\Xi }}\sqrt{k(\xi ,\xi )}\mathbb{P}\left(d\xi \right)<\infty \}$, the MMD between $\mathbb{P},\mathbb{Q}\in {\mathcal{P}}_k\left(\mathrm{\Xi }\right)$ is defined as:

$$\begin{array}{cc}\hfill & {\mathbb{D}}_k(\mathbb{P},\mathbb{Q}):=\underset{f\in {\mathcal{H}}_k,{\parallel f\parallel }_k\le 1}{sup}\left|{\mathbb{E}}_{\mathbb{P}}\left[f\left(\xi \right)\right]-{\mathbb{E}}_{\mathbb{Q}}\left[f\left(\xi \right)\right]\right|.\hfill \end{array}$$

(7)

Dellaporta et al. [20] define the NPL target in (6) as:

$$\begin{array}{c}\hfill {\theta }_k\left({\mathbb{P}}^{\star }\right):=\underset{\theta \in {\mathbb{R}}^k}{arg\; min}{\mathbb{D}}_k({\mathbb{P}}^{\star },{\mathbb{P}}_{\theta }).\end{array}$$

(8)

One of the attractive properties of the MMD is that the supremum in (7) can be obtained in closed form as

$$\begin{array}{cc}\hfill {\mathbb{D}}_k^2(\mathbb{P},\mathbb{Q})& ={\mathbb{E}}_{\xi ,{\xi }^{\prime }\sim \mathbb{P}}\left[k(\xi ,{\xi }^{\prime })\right]-2{\mathbb{E}}_{\xi \sim \mathbb{P},{\xi }^{\prime }\sim \mathbb{Q}}\left[k(\xi ,{\xi }^{\prime })\right]\hfill \\ \hfill & +{\mathbb{E}}_{\xi ,{\xi }^{\prime }\sim \mathbb{Q}}\left[k(\xi ,{\xi }^{\prime })\right]\hfill \end{array}$$

(9)

and can be approximated via sampling (see, e.g., [29]). The resulting NPL posterior with the MMD is called an NPL-MMD posterior.

We explore the gains of using the MMD both in the robust NPL posterior and in the Bayesian ambiguity set. Since the NPL-MMD posterior will target the point in the model family closest (w.r.t. the MMD) to ${\mathbb{P}}^{\star }$, the MMD-based ambiguity set may require a smaller radius to include ${\mathbb{P}}^{\star }$, resulting in less conservative decisions.

2.4. DRO with the Maximum Mean Discrepancy

The MMD has previously been used as a distance metric in the DRO context by Staib and Jegelka [5] who considered the ambiguity set

$${\mathcal{B}}_{ϵ}^k\left({\mathbb{P}}_n\right):=\{\mathbb{P}\in {\mathcal{P}}_k\left(\mathrm{\Xi }\right):{\mathbb{D}}_k(\mathbb{Q},{\mathbb{P}}_n)\le ϵ\}$$

where $ϵ>0$ is the radius of the MMD ball, k is the kernel and ${\mathbb{P}}_n:={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{\delta }_{{\xi }_i}$ is the empirical measure of the observations ${\xi }_{1:n}$. As highlighted by the authors, this ambiguity set has several advantages. First, MMD-DRO can be readily applied to complex data structures such as images or graphs by choosing an appropriate kernel defined on the corresponding data space. This is in contrast to DRO with other choices of distance metrics like the popular Wasserstein distance, in which case, a lot of theoretical and optimisation results rely on specific choices of ground metric, which limit their applicability to more complex data. Moreover, existing finite sample conctration results (see, e.g., [29,30]) can be used for radius selection. Importantly, these results do not suffer from the curse of dimensionality and do not require any assumptions on the DGP, contrary to other distance metrics like the Wasserstein [4]. Although several works have provided remedies to this curse [31], results usually require assumptions on the DGP or the loss function and are hence not always applicable. However, these advantages come at a cost: Ref. [5] explored the optimisation of the MMD-DRO objective by deriving an upper bound which, however, requires for the loss function to be a member of the RKHS, which is often hard to verify. To remedy this, Zhu et al. [6] studied the same ambiguity set along with an extended family of kernel-based ambiguity sets, collectively called Kernel DRO, and provided strong duality results of the optimisation problem which do not require the loss function to be a member of the corresponding RKHS. In this work, we adopt this dual formulation and give a detailed explanation in Section 3.1. Chen et al. [32] generalised this to conditional Kernel DRO, leveraging conditional distributions and Romao et al. [33] explored this framework for dynamic programming.

3. DRO with Robust Bayesian Ambiguity Sets

We propose a robust version of DRO-BAS_PP (3) via the MMD and the NPL-MMD posterior predictive defined as:

$$\begin{array}{c}\hfill {\mathbb{P}}_n^{\mathrm{pred}\left(\mathrm{NPL}\right)}:={\mathbb{E}}_{\mathbb{Q}\sim DP(c^{\prime },{\mathbb{F}}^{\prime })}\left[{\mathbb{P}}_{{\theta }_k\left(\mathbb{Q}\right)}\right].\end{array}$$

(10)

We assume that for $\mathbb{Q}\in \mathcal{P}\left(\mathrm{\Xi }\right)$, the map $\mathbb{Q}\mapsto {\theta }_k\left(\mathbb{Q}\right)$ is measurable such that ${\theta }_k\left(\mathbb{Q}\right)\in arg{min}_{\theta \in \mathrm{\Theta }}{D}_k(\mathbb{Q},{\mathbb{P}}_{\theta })$, so that the expectation in (10) is well defined. Throughout, we also assume that $P_n^{\mathrm{pred}\left(\mathrm{NPL}\right)}\in {\mathcal{P}}_k\left(\mathrm{\Xi }\right)$, which holds, for example, when the kernel k is bounded. A sufficient condition ensuring the existence of the minimiser in (8) is given in Section 3.2. Notice that contrary to the standard Bayesian posterior predictive, ${\mathbb{P}}_n^{\mathrm{pred}\left(\mathrm{NPL}\right)}$ is defined through marginalisation over the nonparametric posterior over the DGP and is defined for any choice of nonparametric prior $DP(\alpha ,\mathbb{F})$ as defined in Section 2.1. Since the MMD can be approximated only via samples (Section 2.3), a closed-form density for the predictive in (10) is not required. We define the following Robust Bayesian Ambiguity Set (RoBAS) with the NPL posterior predictive:

$$\begin{array}{c}\hfill {\mathcal{B}}_{ϵ}^k\left({\mathbb{P}}_n^{\mathrm{pred}\left(\mathrm{NPL}\right)}\right):=\{\mathbb{P}\in {\mathcal{P}}_k\left(\mathrm{\Xi }\right):{\mathbb{D}}_k(\mathbb{P},{\mathbb{P}}_n^{\mathrm{pred}\left(\mathrm{NPL}\right)})\le ϵ\}.\end{array}$$

Note that ${\mathcal{B}}_{ϵ}^k({\mathbb{P}}_n^{\mathrm{pred}\left(\mathrm{NPL}\right)})$ forms a ball when k is a characteristic kernel, as this makes the MMD a probability metric. This property is desirable as it guarantees that the MMD will be zero if and only if $\mathbb{P}\equiv {\mathbb{P}}_n^{\mathrm{pred}\left(\mathrm{NPL}\right)}$. We hence obtain the following DRO-RoBAS worst-case risk problem:

$$\begin{array}{c}\hfill \underset{x\in \mathcal{X}}{min}\underset{{\displaystyle \mathbb{P}\in {\mathcal{B}}_{ϵ}^k({\mathbb{P}}_n^{\mathrm{pred}\left(\mathrm{NPL}\right)})}}{sup}{\mathbb{E}}_{\xi \sim \mathbb{P}}\left[f_x\left(\xi \right)\right]\end{array}$$

(11)

Similarly to DRO-BAS, this optimisation problem corresponds to a worst-case risk over a set of probability measures informed by posterior beliefs. Posterior beliefs about $\theta$ are obtained via posterior beliefs about the DGP and the map ${\theta }_k$ in (8). Since the goal of DRO is to target uncertainty about the DGP, the NPL posterior is a natural choice to inform the ambiguity set as it takes into account any prior beliefs about the DGP. Moreover, by targeting the MMD, rather than the KL divergence as in the BAS case, RoBAS is not restricted to probability measures that are absolutely continuous with respect to ${\mathbb{P}}_{\theta }$.

Intuitively, RoBAS is expected to be a better-informed ambiguity set than BAS when the model is misspecified since it is informed by a robust posterior predictive and a robust discrepancy measure. This is better understood through a toy example. Figure 2 shows a Gaussian location model in the presence of outliers. The top panel shows the DGP of the training data contaminated with 20% of outliers along with a pathological model ${\mathbb{P}}_{\mathrm{pathological}}$ case with a mean larger than that of the DGP. In the BAS_PE case, the expected KL from the DGP to the model is significantly larger than that from the pathological model ${\mathbb{P}}_{\mathrm{pathological}}$ due to the sensitivity of the Bayesian posterior to outliers. A similar result holds for the KL divergence between the posterior predictive and the DGP and pathological model in the BAS_PP case. In contrast, the MMD from the NPL posterior predictive to the DGP is much smaller compared to that of the pathological model. Alternative DRO-RoBAS formulations are provided in Appendix C. The results of this section are in Appendix A.

3.1. Duality of the DRO-RoBAS  Problem

We first formulate our optimisation problem as a kernel DRO problem [6]. This allows us to obtain a dual formulation of (11) in the RKHS which can be optimised using kernel methods. Let $\varphi :\mathrm{\Xi }\to {\mathcal{H}}_k$ denote the feature map associated with kernel k and let ${\mu }_{\mathbb{P}}\in {\mathcal{H}}_k$ denote the kernel mean embedding of the probability measure $\mathbb{P}\in {\mathcal{P}}_k\left(\mathrm{\Xi }\right)$, i.e., ${\mu }_{\mathbb{P}}:={\mathbb{E}}_{\xi \sim \mathbb{P}}\left[\varphi \left(\xi \right)\right]$. Then the MMD is equivalently defined as ${\mathbb{D}}_k(\mathbb{P},\mathbb{Q})={\parallel {\mu }_{\mathbb{P}}-{\mu }_{\mathbb{Q}}\parallel }_k$ ([29], Lemma 4). Throughout we assume that ${\mathbb{P}}_n^{\mathrm{pred}\left(\mathrm{NPL}\right)}\in {\mathcal{P}}_k\left(\mathrm{\Xi }\right)$. Note that this condition is trivially satisfied for a bounded kernel k. Consider the following set satisfying the conditions of RoBAS:

$$\begin{array}{c}\hfill {\mathcal{C}}^{\star }:=\{\mu \in {\mathcal{H}}_k:\parallel \mu -{\mu }_{{\mathbb{P}}_n^{\mathrm{pred}\left(\mathrm{NPL}\right)}}{\parallel }_k\le ϵ\}.\end{array}$$

(12)

The associated ambiguity set induced by ${\mathcal{C}}^{\star }$ is

$$\begin{array}{c}\hfill \begin{array}{cc}\hfill {\mathcal{K}}_{{\mathcal{C}}^{\star }}& :=\{\mathbb{P}\in {\mathcal{P}}_k\left(\mathrm{\Xi }\right),{\mu }_{\mathbb{P}}\in {\mathcal{C}}^{\star }\}\equiv {\mathcal{B}}_{ϵ}^k({\mathbb{P}}_n^{\mathrm{pred}\left(\mathrm{NPL}\right)})\hfill \end{array}\end{array}$$

(13)

and our DRO-RoBAS problem in (11) is equivalent to:

$$\begin{array}{c}\begin{array}{c}\hfill \underset{x\in \mathcal{X}}{min}\underset{\mathbb{P},{\mu }_{\mathbb{P}}}{sup}{\mathbb{E}}_{\xi \sim \mathbb{P}}\left[f_x\left(\xi \right)\right]\mathrm{s}.\mathrm{t}.\mathbb{P}\in {\mathcal{P}}_k\left(\mathrm{\Xi }\right),{\mu }_{\mathbb{P}}\in {\mathcal{C}}^{\star }.\end{array}\end{array}$$

(14)

The equivalence can be seen through the ${\mathcal{C}}^{\star }$-induced ambiguity set for distributions $\mathbb{P}$ which can be written as ${\mathcal{K}}_{{\mathcal{C}}^{\star }}$.

Before we proceed to our dual formulation, we introduce the main result from Zhu et al. [6] which gives a dual formulation of Kernel DRO problems for general sets $\mathcal{C}$ satisfying certain assumptions.

Theorem 1 

(Zhu et al. [6], Theorem 3.1). Assume $\mathcal{C}\subset {\mathcal{H}}_k$ is closed convex, $f_x(·)$ is proper, upper semi-continuous, and $ri\left({\mathcal{K}}_{\mathcal{C}}\right)\ne \varnothing$, where $ri\left({\mathcal{K}}_{\mathcal{C}}\right)$ denotes the relative interior of ${\mathcal{K}}_{\mathcal{C}}$. Then the primal problem:

$$\begin{array}{cc}\hfill & \underset{x}{min}\underset{\mathbb{P},\mu }{sup}{\mathbb{E}}_{\xi \sim \mathbb{P}}\left[f_x\left(\xi \right)\right]s.t.\mathbb{P}\in \mathcal{P},{\mu }_{\mathbb{P}}=\mu ,\mu \in \mathcal{C}\hfill \end{array}$$

is equivalent to:

$$\begin{array}{c}\hfill \underset{x,g_0\in \mathbb{R},g\in {\mathcal{H}}_k}{min}{g}_0+{\delta }_{\mathcal{C}}^{\star }\left(g\right)s.t.f_x\left(\xi \right)\le g_0+g\left(\xi \right),\forall \xi \in \mathrm{\Xi }\end{array}$$

where ${\delta }_{\mathcal{C}}^{\star }\left(g\right):={sup}_{\mu \in \mathcal{C}}{〈g,\mu 〉}_{{\mathcal{H}}_k}$ the support function of $\mathcal{C}$.

This theorem gives an effective way to transition from the primal to the dual formulation by using the support function of the set $\mathcal{C}$. Importantly, in contrast to other kernel-based dual formulations (e.g., [5]), this theorem does not require the objective function f to be a member of the RKHS ${\mathcal{H}}_k$. We first derive the support function of ${\mathcal{C}}^{\star }$ in the DRO-RoBAS case. We denote ${\mathbb{E}}_{\mathbb{Q}\sim DP({\alpha }^{\prime },{\mathbb{F}}^{\prime })}$ and ${\mathbb{E}}_{\xi \sim {\mathbb{P}}_{{\theta }_k\left(\mathbb{Q}\right)}}$ by ${\mathbb{E}}_{{DP}_{{\xi }_{1:n}}}$ and ${\mathbb{E}}_{{\mathbb{P}}_{{\theta }_k\left(\mathbb{Q}\right)}}$ respectively.

Proposition 1. 

Let ${\mathcal{C}}^{\star }$ be defined by (12). Then we have ${\delta }_{{\mathcal{C}}^{\star }}^{\star }\left(g\right)={\mathbb{E}}_{{DP}_{{\xi }_{1:n}}}\left[{\mathbb{E}}_{{\mathbb{P}}_{{\theta }_k\left(\mathbb{Q}\right)}}\left[g\left(\xi \right)\right]\right]+ϵ{\parallel g\parallel }_k$.

We can now apply Theorem 1 to our problem.

Corollary 1. 

Let ${\mathcal{C}}^{\star }$ as in (12) and $f_x(·)$ proper, upper semi-continuous. Then problem (14) is equivalent to:

$$\begin{array}{c}\begin{array}{cccc}& \underset{x,g_0\in \mathbb{R},g\in {\mathcal{H}}_k}{min}\hfill & & g_0+{\mathbb{E}}_{{DP}_{{\xi }_{1:n}}}\left[{\mathbb{E}}_{{\mathbb{P}}_{{\theta }_k\left(\mathbb{Q}\right)}}\left[g\left(\xi \right)\right]\right]+ϵ{\parallel g\parallel }_k\hfill \\ & \mathrm{subject}\mathrm{to}\hfill & & f_x\left(\xi \right)\le g_0+g\left(\xi \right),\forall \xi \in \mathrm{\Xi }.\hfill \end{array}\end{array}$$

(15)

• Computation of (15):

The problem in (15) can be solved by the batch approach with discretization of a semi-infinite programme (SIP) [35] suggested in Zhu et al. [6], in addition to a Sample Average Approximation (SAA). Let ${\left\{{\widehat{\xi }}_i\right\}}_{i=1}^N$ be samples from the nested expectation in (15) and ${\left\{{\zeta }_j\right\}}_{j=1}^m\subseteq {\mathrm{\Xi }}^m$ be a set of discretisation points. Then the problem can be approximated by:

$$\begin{array}{c}\begin{array}{cccc}& \underset{x,g_0\in \mathbb{R},g\in {\mathcal{H}}_k}{min}\hfill & & g_0+{\displaystyle \frac{1}{N}}\sum _{i=1}^{N}g\left({\widehat{\xi }}_i\right)+ϵ{\parallel g\parallel }_k\hfill \\ & \mathrm{s}.\mathrm{t}.\hfill & & f_x\left({\zeta }_j\right)\le g_0+g\left({\zeta }_j\right),\forall j\in \left[m\right].\hfill \end{array}\end{array}$$

(16)

We can now apply the distributional robust version of the Representer theorem ([6], Lemma B.1) which states that it is sufficient to parametrise g by $g(·)={\sum }_{i=1}^N{\alpha }_{i}k({\widehat{\xi }}_i,·)+{\sum }_{j=1}^m{\alpha }_{N+j}k({\zeta }_j,·)$ for some ${\alpha }_k\in \mathbb{R}$, for all $k=1,\dots ,N+m$.

3.2. Tolerance Level Guarantees

We start by using the generalisation error results for the NLP-MMD posterior to obtain a result in probability that the DGP lies within our ambiguity set. First, we give a concentration type bound for ${\mathbb{E}}_{\mathbb{Q}\sim DP({\alpha }^{\prime },{\mathbb{F}}^{\prime })}\left[{\mathbb{D}}_k({\mathbb{P}}^{\star },\mathbb{Q})\right]$. In practice, exact sampling from a DP is not possible, so we consider the approximation of the DP suggested in the NPL literature [15,16,20] to sample during the MMD Posterior Bootstrap. In particular, denote by ${\widehat{DP}}_{{\xi }_{1:n}}$ the probability measure on $\mathcal{P}\left(\mathrm{\Xi }\right)$ induced by the following sampling process for $(w_{1:n},{\tilde{w}}_{1:\tau })\sim Dir\left(1,\dots ,1,{\displaystyle \frac{\alpha }{\tau }},\dots ,{\displaystyle \frac{\alpha }{\tau }}\right)$ and ${\tilde{\xi }}_{1:\tau }\stackrel{\mathrm{iid}}{\sim }\mathbb{F}$:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill \mathbb{Q}:=\sum _{i=1}^n{w}_i{\delta }_{{\xi }_i}+\sum _{k=1}^{\tau }{\tilde{w}}_k{\delta }_{{\tilde{\xi }}_k}\sim {\widehat{DP}}_{{\xi }_{1:n}}.\end{array}\end{array}$$

(17)

The associated approximate posterior predictive is ${\widehat{\mathbb{P}}}_n^{\mathrm{pred}\left(\mathrm{NPL}\right)}:={\mathbb{E}}_{\mathbb{Q}\sim {\widehat{DP}}_{{\xi }_{1:n}}}\left[{\mathbb{P}}_{{\theta }_k\left(\mathbb{Q}\right)}\right].$ We provide a concentration result for the MMD between ${\mathbb{P}}^{\star }$ and ${\widehat{\mathbb{P}}}_n^{\mathrm{pred}\left(\mathrm{NPL}\right)}$ as the approximated predictive is used in practice. However, similar results can be derived for the exact case with ${\mathbb{P}}_n^{\mathrm{pred}\left(\mathrm{NPL}\right)}$ following the arguments in [20]. Additionally, all theoretical results regarding the duality of the DRO-RoBAS framework from Section 3.1 hold exactly the same for the approximated DP as they are proven for a general posterior. We make the following assumptions:

Assumption 1. 

For every $\mathbb{Q}\in \mathcal{P}\left(\mathrm{\Xi }\right)$ there exists $c>0$ such that the set $\{\theta \in \mathrm{\Theta }:{\mathbb{D}}_k(\mathbb{Q},{\mathbb{P}}_{\theta })\le {inf}_{\theta \in \mathrm{\Theta }}{\mathbb{D}}_k(\mathbb{Q},{\mathbb{P}}_{\theta })+c\}$ is bounded.

Assumption 2. 

The kernel k is such that $\left|k\right(\xi ,{\xi }^{\prime }\left)\right|\le M$, $M<\infty$, for any $\xi ,{\xi }^{\prime }\in \mathrm{\Xi }$.

Assumption 1 ensures that a minimiser in (8) exists and is a common assumption made in MMD estimator methods (see [30]). Assumption 2 is needed to obtain a concentration inequality for the NPL posterior and it is often made in methods using MMD estimators (see, e.g., [20,30,36,37]), as it ensures robustness guarantees (see, e.g., [20,30,36,37]). Intuitively, bounded kernels control the contribution of extreme or unlikely observations, preventing them from having an arbitrarily large effect on the distance and thereby yielding a robust measure of discrepancy. Many commonly used kernels are bounded, such as the Gaussian, Matern and Exponential kernels (see, e.g., [38]).

Theorem 2. 

Suppose Assumptions 1 and 2 hold. Then with probability at least $1-\delta$:

$$\begin{array}{c}\hfill \begin{array}{c}\hfill {\mathbb{D}}_k({\mathbb{P}}^{\star },{\widehat{\mathbb{P}}}_n^{\mathrm{pred}\left(\mathrm{NPL}\right)})\le \underset{\theta \in \mathrm{\Theta }}{inf}{\mathbb{D}}_k({\mathbb{P}}_{\theta },{\mathbb{P}}^{\star })+C_{n,M,\alpha }\end{array}\end{array}$$

(18)

where $C_{n,M,\alpha }$ is a constant depending on the number of samples n, the upper bound of the kernel M and the concentration parameter on the DP prior α.

Remark 1. 

$C_{n,M,\alpha }$ has an overall rate of $1/\sqrt{n}$ consistent with existing results for minimum MMD estimators [30,36]. Moreover, given an upper bound of the kernel M, the constant is fully known. Hence, if ${inf}_{\theta \in \mathrm{\Theta }}{\mathbb{D}}_k({\mathbb{P}}_{\theta },{\mathbb{P}}^{\star })$ can be reasonably approximated, this result can be used to select the radius ensuring RoBAS includes ${\mathbb{P}}^{\star }$ with high probability. However, in practice, this theoretical radius can be over-conservative and a suitable value can be chosen via cross-validation or related bootstrapping procedures following standard practices in the DRO literature (see, e.g., [12,39,40]).

We can now obtain an upper bound for the target optimisation problem for large enough $ϵ$.

Corollary 2. 

Suppose Assumptions 1 and 2 hold and let $C_{n,M,\alpha }$ as in Theorem 2. Then, for $ϵ\ge C_{n,M,\alpha }+{inf}_{\theta \in \mathrm{\Theta }}{\mathbb{D}}_k({\mathbb{P}}_{\theta },{\mathbb{P}}^{\star })$, with probability at least $1-\delta$:

$$\begin{array}{cc}\hfill & {\mathbb{E}}_{\xi \sim {\mathbb{P}}^{\star }}\left[f_x\left(\xi \right)\right]\le \underset{{\mathcal{B}}_{ϵ}^k\left({\widehat{\mathbb{P}}}_n^{\mathrm{pred}\left(\mathrm{NPL}\right)}\right)}{sup}{\mathbb{E}}_{\xi \sim \mathbb{P}}\left[f_x\left(\xi \right)\right].\hfill \end{array}$$

In the special case of Huber’s contamination model Huber [41], we can obtain a guarantee similar to Theorem 2 which depends on the contamination level.

Corollary 3 

(Huber’s cont. model). Suppose ${\mathbb{P}}^{\star }=(1-\eta ){\mathbb{P}}_{{\theta }_0}+\eta \mathbb{Q}$ for some ${\theta }_0\in \mathrm{\Theta }$, $\mathbb{Q}\in \mathcal{P}\left(\mathrm{\Xi }\right)$ and $\eta \in [0,1]$. Suppose Assumptions 1 and 2 hold and let $C_{n,M,\alpha }$ as in Theorem 2. Then with probability at least $1-\delta$: ${\mathbb{D}}_k({\mathbb{P}}_{{\theta }_0},{\widehat{\mathbb{P}}}_n^{\mathrm{pred}\left(\mathrm{NPL}\right)})\le 4\eta +2C_{n,M,\alpha }.$

4. Experiments

We evaluate our method on several different DGPs, model families and misspecification settings for two decision-making problems: the Newsvendor and the Portfolio.

We compare our method to existing Bayesian formulations of DRO—DRO-BAS [12] and Bayesian DRO (BDRO) [11]—both based on the KL divergence and standard Bayesian posterior. To assess how much robustness in our framework is gained through the robust posterior compared to the choice of the MMD in the ambiguity set, we further compare against the empirical method which uses an MMD ball around the empirical measure (denoted by Empirical MMD). This was presented in Staib and Jegelka [5] and also forms a special case of Kernel DRO [6]. Implementation details are provided in Appendix B.

We explore two types of misspecification.

• Model misspecification which occurs when the DGP ${\mathbb{P}}^{\star }$ does not belong to the model family ${\mathcal{P}}_{\mathrm{\Theta }}$, e.g., if ${\mathbb{P}}^{\star }$ is multimodal while ${\mathcal{P}}_{\mathrm{\Theta }}$ assumes unimodality. This affects Bayesian DRO methods (BDRO, DRO-BAS, DRO-RoBAS) but not empirical approaches, as the latter do not rely on a model.
• Huber’s contamination model [41] which is a specific type of model misspecification (see Figure 2) wherein the training DGP is ${\mathbb{P}}^{\star }=(1-\eta )\tilde{\mathbb{P}}+\eta \mathbb{Q}$ for some $\eta \in [0,1]$ and $\tilde{\mathbb{P}},\mathbb{Q}\in \mathcal{P}\left(\mathrm{\Xi }\right)$. Contamination, limited to the training set, impacts both Bayesian and empirical DRO methods since the test distribution is assumed to be $\tilde{\mathbb{P}}$. Huber contamination relates to concepts like distribution shift and out-of-distribution robustness (e.g., [42]) and its importance in DRO has attracted increasing attention in recent work [43,44].

4.1. The Newsvendor Problem

We start with the commonly explored Newsvendor problem (e.g., [45]). The goal is to choose the optimal amount of products to buy based on consumers’ demand. The cost is defined as: $f(x,\xi ):=hmax(x-\xi ,0)+bmax(\xi -x,0)$ where $x\in {\mathbb{R}}_{\ge 0}^D$ is the number of product units ordered, $\xi \in {\mathbb{R}}^D$ is the consumers’ demand, $0\in {\mathbb{R}}^D$ is the zero vector and b and h denote the backorder and holding cost per unit respectively. In all examples we follow the implementation of Shapiro et al. [11], Dellaporta et al. [12] and set $b=8$ and $h=3$. We run each experiment $J=100$ times for $n=20$ observations and compute the out-of-sample mean and variance of the cost incurred.

We consider two models and several DGP cases as follows. First, we assume the demand $\xi \in \mathrm{\Xi }$ follows a Gaussian distribution with known variance, i.e., ${\mathbb{P}}_{\theta }:=\mathcal{N}(\theta ,{\sigma }^2{\mathcal{I}}_{D\times D})$ while the DGP is a bimodal Gaussian distribution (case 1 above): ${\mathbb{P}}^{\star }:=0.5\mathcal{N}({\theta }_1^{\star },{\sigma }^2{\mathcal{I}}_{D\times D})+0.5\mathcal{N}({\theta }_2^{\star },{\sigma }^2{\mathcal{I}}_{D\times D})$ for $D=1$ and $D=5$. Furthermore, we explore the same Gaussian model with a contaminated Gaussian training DGP (case 2 above): ${\mathbb{P}}_{\mathrm{train}}^{\star }:=(1-\eta )\mathcal{N}({\theta }^{\star },{\sigma }^2)+\eta \mathcal{N}({\theta }^{\prime },{\sigma }^2)$ and an Exponential model ${\mathbb{P}}_{\theta }:=\mathrm{Exp}\left(\theta \right)$ with a contaminated Exponential DGP for the training DGP (case 2 above): ${\mathbb{P}}_{\mathrm{train}}^{\star }:=(1-\eta )\mathrm{Exp}\left({\theta }^{\star }\right)+\eta \mathcal{N}(\mu ,\sigma )$ for $\eta \in \{0.0,0.1,0.2\}$.

Figure 3 presents the out-of-sample mean and variance of the methods for the bimodal univariate and multivariate Gaussian DGPs.

The effect of model misspecification is notably more pronounced for the DRO-BAS and BDRO instantiations, which are based on the standard Bayesian posterior and the KL divergence. In the DRO-RoBAS case, this robustness is likely due to the fact that the obtained NPL-MMD posterior is bimodal, even though the model itself is unimodal. In contrast, the standard Bayesian posterior is highly sensitive to misspecification, resulting in a unimodal posterior concentrated between the two modes. Consequently, DRO-BAS and BDRO require very large values of $ϵ$ to capture the true DGP, leading to conservative decisions that incur high out-of-sample costs. This is evident as they achieve lower mean-variance as $ϵ$ increases.

We further observe (Figure 3) that in the univariate case, empirical MMD achieves a lower (difference of ≤1) out-of-sample mean for most values of $ϵ<1$. However, DRO-RoBAS consistently shows a lower out-of-sample variance. In the multivariate case, the performances of the two methods are similar, though empirical MMD outperforms DRO-RoBAS in both mean and variance. Notably, this comparison pits a Bayesian method under model misspecification against a completely empirical method unaffected by this misspecification.

However, it is promising that DRO-RoBAS remains highly competitive against this baseline. The next example, based on contamination models, illustrates a scenario where robustness is crucial for both model-based and empirical methods.

In the second simulation, we consider the Huber contamination models [41] where the training set is contaminated, whereas the test set is not. Figure 4 demonstrates that both DRO-BAS formulations outperform DRO-RoBAS and the empirical MMD method in the well-specified case, where there is no contamination in the training set, and the training and test distributions are identical. In misspecified cases, where $\eta >0$, DRO-RoBAS shows greater robustness compared to the other methods in terms of the out-of-sample mean-variance trade-off. This suggests that the robust posterior and robust distance measures in RoBAS contribute to a better-informed ambiguity set concerning the test set generating process.

This example further illustrates that, while the motivation for a robust ambiguity set stemmed from concerns about model misspecification, even entirely empirical methods, like the empirical MMD, can be sensitive to misspecifications arising from discrepancies between the training and test distributions.

4.2. The Portfolio Optimisation Problem

We continue with the multi-dimensional Portfolio problem, also considered by Shapiro et al. [11], which chooses stock weightings ($x\in {\mathbb{R}}^D$) to maximise returns. The objective function is $f_x\left(\xi \right)=-{\xi }^{\top }x$ which corresponds to maximising the return and the optimisation is subject to the constraints $x_i\ge 0$ for all $i=1,\dots ,D$ and ${\sum }_{i=1}^D{x}_i=1$. We generate $n=100$ observations from a 5D Gaussian DGP with contamination on three-out-of-five dimensions: ${\mathbb{P}}^{\star }=(1-\eta )\mathcal{N}({\mu }^{\star },{\Sigma }^{\star })+\eta \mathcal{N}({\mu }^{\prime },{\Sigma }^{\star })$. We use a multivariate Gaussian model with unknown mean and variance.

Figure 5 shows DRO-RoBAS is unaffected by the contamination, whilst empirical MMD, DRO-BAS, and BDRO are negatively affected. Consider $\eta =0.1$: for $ϵ<0.2$, empirical DRO performance quickly degrades compared to DRO-RoBAS; but, for $ϵ\ge 0.2$, empirical MMD performs similarly due to the MMD robustness. This effect is magnified for $\eta =0.2$, demonstrating that the empirical nominal distribution is unreliable due to outliers, whilst DRO-RoBAS benefits from a robust nominal—the NPL-MMD posterior predictive—thus performs better for small $ϵ$.

4.3. Computational Time

The increased robustness of DRO-RoBAS comes at the cost of higher computational demands (see

Table A1. Average (standard derivation) solve time in seconds of algorithms across different DGPs.

DGP	RoBAS	Empirical MMD	KL-BDRO	DRO-BASPE	DRO-BASPP
1D $\mathcal{N}$ bimodal	275.33 (25.98)	2.17 (0.25)	3.90 (0.55)	0.29 (0.03)	0.30 (0.05)
5D $\mathcal{N}$ bimodal	262.94 (31.72)	2.18 (0.36)	2.63 (0.26)	1.24 (0.19)	1.38 (0.29)
$\mathcal{N},\eta =0.0$	286.27 (16.14)	2.22 (0.28)	4.01 (0.49)	0.29 (0.04)	0.30 (0.04)
$\mathcal{N},\eta =0.1$	284.62 (17.42)	2.23 (0.24)	3.99 (0.42)	0.29 (0.03)	0.30 (0.04)
$\mathcal{N},\eta =0.2$	260.67 (35.90)	2.20 (0.24)	3.74 (0.52)	0.28 (0.03)	0.28 (0.04)
Exp, $\eta =0.0$	311.74 (37.48)	5.59 (0.68)	0.43 (0.06)	0.40 (0.06)	0.49 (0.06)
Exp, $\eta =0.1$	296.65 (46.10)	5.39 (0.54)	0.44 (0.06)	0.42 (0.05)	0.50 (0.06)
Exp, $\eta =0.2$	282.55 (53.23)	5.22 (0.47)	0.44 (0.06)	0.42 (0.05)	0.50 (0.06)

and

Table A2. Average (standard derivation) sample time in seconds of algorithms across different DGPs.

DGP	RoBAS	Empirical MMD	KL-BDRO	DRO-BASPE	DRO-BASPP
1D $\mathcal{N}$ bimodal	15.65 (4.76)	0.0 (0.0)	0.001 (0.002)	0.0002 (0.0)	0.0002 (0.0)
5D $\mathcal{N}$ bimodal	15.86 (4.84)	0.0 (0.0)	0.0104 (0.0004)	0.0006 (0.0003)	0.001 (0.0004)
$\mathcal{N},\eta =0.0$	15.85 (4.67)	0.0 (0.0)	0.0009 (0.002)	0.0002 (0.0)	0.0002 (0.0)
$\mathcal{N},\eta =0.1$	15.78 (4.59)	0.0 (0.0)	0.0009 (0.0017)	0.0002 (0.0)	0.0002 (0.0)
$\mathcal{N},\eta =0.2$	15.68 (4.58)	0.0 (0.0)	0.0008 (0.0014)	0.0002 (0.0)	0.0002 (0.0)
Exp, $\eta =0.0$	38.10 (5.62)	0.0 (0.0)	0.1 (0.01)	0.0 (0.0)	0.0 (0.0)
Exp, $\eta =0.1$	37.77 (4.78)	0.0 (0.0)	0.1 (0.01)	0.0 (0.0)	0.0 (0.0)
Exp, $\eta =0.2$	37.82 (4.85)	0.0 (0.0)	0.1 (0.01)	0.0 (0.0)	0.0 (0.0)

of Appendix B). This cost arises from the complex optimisation problem in the RKHS and the longer sampling time required for the NPL posterior. However, as previously demonstrated, this cost is justified by improved out-of-sample performance across various cases of model misspecification. Moreover, by leveraging the NPL-MMD and the MMD, DRO-RoBAS can be used for any choice of model family, even likelihood-free models. Notice that DRO-BAS_PE is limited to Exponential family models whereas the computational cost of BDRO and DRO-BAS_PP increases considerably if the posterior is not available in closed form, as methods like Markov Chain Monte Carlo are needed for posterior sampling. This highlights the flexibility and robustness of DRO-RoBAS despite its computational demands. Nevertheless, possible scalability improvements are discussed in Section 5.

5. Conclusions

Bayesian formulations of DRO for decision-making problems can suffer from model misspecification as the ambiguity set heavily relies on the non-robust Bayesian posterior. We addressed this challenge by using a robust NPL posterior to inform the ambiguity set and leveraging the MMD to construct both the posterior and the ambiguity set itself. We show that DRO-RoBAS admits a dual formulation in the RKHS and we provide probabilistic guarantees for the tolerance level such that the resulting optimisation problem upper bounds the true objective with high probability.

Scalability improvements for DRO-RoBAS can be achieved through existing tools from kernel methods such as Fourier features [46] and low-rank kernel matrix approximations [47] that are left for future work. Our empirical evidence suggests that if the model is well specified, or the level of misspecification is low, then existing Bayesian formulations like DRO-BAS can have better performance and scalability. At the same time, when model misspecification is moderate or high then DRO-RoBAS achieves significantly improved out-of-sample performance and robustness. Note that any prior knowledge on the misspecification level can be naturally incorporated into our framework to further boost performance.

Finally, the construction of Robust Bayesian Ambiguity Sets can extend beyond the choices of NPL and MMD. For example, another instantiation of our framework arises if we employ Generalised Bayesian Inference (GBI) [21]. One of the motivations of GBI is to induce robustness with respect to model misspecification [22,23,24,25,26,48,49] by targeting a different divergence than the KL. While these methods do not directly impose uncertainty on the DGP like NPL does, they can produce robust GBI posteriors, making it worthwhile to integrate into DRO-RoBAS. Notably, the duality results in Section 3.1 are based on the MMD choice; however, they hold under a general posterior and are not dependent on the NPL framework.