The Learning Rate Is Not a Constant: Sandwich-Adjusted Markov Chain Monte Carlo Simulation

Abstract

A fundamental limitation of maximum likelihood and Bayesian methods under model misspecification is that the asymptotic covariance matrix of the pseudo-true parameter vector ${\mathsf{\theta }}_{*}$ is not the inverse of the Fisher information, but rather the sandwich covariance matrix ${\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{-1}{\mathbf{B}}_{*}^{\phantom{-1}}{\mathbf{A}}_{*}^{-1}$, where ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$ are the sensitivity and variability matrices, respectively, evaluated at ${\mathsf{\theta }}_{*}$ for training data record ${\omega }_1,\dots ,{\omega }_n$. This paper makes three contributions. First, we review existing approaches to robust posterior sampling, including the open-faced sandwich adjustment and magnitude- and curvature-adjusted Markov chain Monte Carlo (MCMC) simulation. Second, we introduce a new sandwich-adjusted MCMC method. Unlike existing approaches that rely on arbitrary matrix square roots, eigendecompositions or a single scaling factor applied uniformly across the parameter space, our method employs a parameter-dependent learning rate $\lambda \left(\mathsf{\theta }\right)$ that enables direction-specific tempering of the likelihood. This allows the sampler to capture directional asymmetries in the sandwich distribution, particularly under model misspecification or in small-sample regimes, and yields credible regions that remain valid when standard Bayesian inference underestimates uncertainty. Third, we propose information-theoretic diagnostics for quantifying model misspecification, including a strictly proper divergence score and scalar summaries based on the Frobenius norm, Earth mover’s distance, and the Herfindahl index. These principled diagnostics complement residual-based metrics for model evaluation by directly assessing the degree of misalignment between the sensitivity and variability matrices, ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$. Applications to two parametric distributions and a rainfall-runoff case study with the Xinanjiang watershed model show that conventional Bayesian methods systematically underestimate uncertainty, while the proposed method yields asymptotically valid and robust uncertainty estimates. Together, these findings advocate for sandwich-based adjustments in Bayesian practice and workflows.

1. Introduction

Suppose that we have a vector-valued statistical (mathematical) model $\mathbf{y}=\mathbf{f}\left(\mathsf{\theta }\right):{\mathbb{R}}^d\to {\mathbb{R}}^n$ of a $d\times 1$-vector of parameters $\mathsf{\theta }={({\theta }_1,\dots ,{\theta }_d)}^{\top }$ that we wish to estimate from training data ${\mathsf{\omega }}_n={({\omega }_1,\dots ,{\omega }_n)}^{\top }$. Common practice is to define a residual loss function $e_t\left(\mathsf{\theta }\right)={\omega }_t-y_t\left(\mathsf{\theta }\right)$ for all $t=1,\dots ,n$ and minimize (maximize, if appropriate) this function using an automatic search algorithm. In this special issue on Bayesian estimation and information theory, we shall use a likelihood function $L_n\left(\mathsf{\theta }\right)$ for $\mathsf{\theta }$ given the n observations ${\omega }_1,\dots ,{\omega }_n$. However, the problem we address in this paper is not limited to Bayesian methods but applies equally to frequentist inference using least-squares methods. We use $L_{\omega }\left(\mathsf{\theta }\right)$ as shorthand notation for $L(\omega \mid \mathsf{\theta })$ and write ${\mathcal{L}}_{\omega }\left(\mathsf{\theta }\right)=log\left(L_{\omega }\left(\mathsf{\theta }\right)\right)$ for the log-likelihood function. The joint likelihood $L_n\left(\mathsf{\theta }\right)$ for the n-vector of data points, ${\omega }_1,\dots ,{\omega }_n$, is equal to the product of $L_{{\omega }_1}\left(\mathsf{\theta }\right),\dots ,L_{{\omega }_n}\left(\mathsf{\theta }\right)$. Now, the unnormalized d-variate posterior density $P_n\left(\mathsf{\theta }\right)=P(\mathsf{\theta }\mid {\mathsf{\omega }}_n)$ follows from Bayes’ theorem [1], $P_n\left(\mathsf{\theta }\right)\propto P\left(\mathsf{\theta }\right)L_n\left(\mathsf{\theta }\right)$, where $P\left(\mathsf{\theta }\right)$ is the prior density. In logarithmic form, ${\mathcal{P}}_n\left(\mathsf{\theta }\right)=\mathcal{P}\left(\mathsf{\theta }\right)+{\mathcal{L}}_n\left(\mathsf{\theta }\right)-log\left(Z_n\right)$, where $\mathcal{P}\left(\mathsf{\theta }\right)=log\left(P\left(\mathsf{\theta }\right)\right)$ is the log-prior and $Z_n=\int P\left(\mathsf{\theta }\right)L_n\left(\mathsf{\theta }\right)\mathrm{d}\mathsf{\theta }$ denotes the marginal likelihood.

The Bernstein and von Mises [2] theorem establishes that when the sample size n grows, the posterior distribution of the parameters becomes approximately normal, centered on the true parameter values ${\mathsf{\theta }}_0$ of the data-generating process and with a covariance matrix ${\displaystyle \frac{1}{n}}{\mathcal{I}}_0^{-1}\left({\mathsf{\theta }}_0\right)$ equal to the inverse of the $d\times d$ Fisher [3] information matrix [4]

$$\begin{array}{cc}\hfill & {\mathcal{I}}_0\left({\mathsf{\theta }}_0\right)={\mathbb{E}}_{\omega }[\nabla {\mathcal{L}}_{\omega }\left({\mathsf{\theta }}_0\right){\left(\nabla {\mathcal{L}}_{\omega }\left({\mathsf{\theta }}_0\right)\right)}^{\top }].\hfill \end{array}$$

(1)

This theorem establishes that Bayesian credible sets asymptotically approximate optimal frequentist confidence sets and, as such, it forms the basis for using Bayesian credible sets in statistical inference. The fundamental underpinning of this theory is the information identity ${\mathbf{A}}_0={\mathbf{B}}_0$, where

$$\begin{array}{cc}\hfill & {\mathbf{A}}_0=-{\mathbb{E}}_{\omega }\left[{\nabla }^2{\mathcal{L}}_{\omega }\left({\mathsf{\theta }}_0\right)\right],\hfill \end{array}$$

(2)

is the so-called sensitivity (negative Hessian) matrix, and

$$\begin{array}{cc}\hfill {\mathbf{B}}_0& ={\mathbb{E}}_{\omega }[\nabla {\mathcal{L}}_{\omega }\left({\mathsf{\theta }}_0\right){\left(\nabla {\mathcal{L}}_{\omega }\left({\mathsf{\theta }}_0\right)\right)}^{\top }]\hfill \\ \hfill & ={Var}_{\omega }[\nabla {\mathcal{L}}_{\omega }\left({\mathsf{\theta }}_0\right)],\hfill \end{array}$$

(3)

is the variability matrix at ${\mathsf{\theta }}_0$. The term “variability” reflects the well-known identity $Var\left[\mathbf{X}\right]=\mathbb{E}\left[(\mathbf{X}-\mathsf{\mu }){(\mathbf{X}-\mathsf{\mu })}^{\top }\right]$ with $\mathsf{\mu }=\mathbb{E}\left[\mathbf{X}\right]$, applied to the score $\nabla {\mathcal{L}}_{\omega }\left({\mathsf{\theta }}_0\right)$

$$\begin{array}{cc}\hfill Var[\nabla {\mathcal{L}}_{\omega }\left({\mathsf{\theta }}_0\right)]& ={\mathbb{E}}_{\omega }\left[\left(\nabla {\mathcal{L}}_{\omega }\left({\mathsf{\theta }}_0\right)-{\mathsf{\mu }}_0\right){\left(\nabla {\mathcal{L}}_{\omega }\left({\mathsf{\theta }}_0\right)-{\mathsf{\mu }}_0\right)}^{\top }\right]\hfill \\ \hfill & ={\mathbb{E}}_{\omega }\left[\nabla {\mathcal{L}}_{\omega }\left({\mathsf{\theta }}_0\right){\left(\nabla {\mathcal{L}}_{\omega }\left({\mathsf{\theta }}_0\right)\right)}^{\top }\right]-{\mathsf{\mu }}_0{\mathsf{\mu }}_0^{\top }.\hfill \end{array}$$

(4)

Under correct specification, the expected score ${\mathsf{\mu }}_0={\mathbb{E}}_{\omega }[\nabla {\mathcal{L}}_{\omega }\left({\mathsf{\theta }}_0\right)]$, equals a zero vector, and, consequently we yield that ${\mathbf{B}}_0={Var}_{\omega }[\nabla {\mathcal{L}}_{\omega }\left({\mathsf{\theta }}_0\right)]$.

We then also have that the maximum likelihood (ML) density estimator ${\widehat{\mathsf{\theta }}}_n$ of the posterior parameter distribution satisfies [5]

$$\begin{array}{cc}\hfill \sqrt{n}({\widehat{\mathsf{\theta }}}_n-{\mathsf{\theta }}_0)& \stackrel{d}{⟶}{\mathcal{N}}_d\left(\mathbf{0},{\mathcal{I}}_0^{-1}\left({\mathsf{\theta }}_0\right)\right),\hfill \end{array}$$

where ${\mathcal{I}}_0\left({\mathsf{\theta }}_0\right)$ is the expected Fisher information for a single datum. Fisher information plays a fundamental role in statistical inference, including hypothesis testing, regression analysis, and the calculation of standard errors and parameter confidence intervals and regions. The information or second Bartlett [6] identity ${\mathbf{A}}_0={\mathbf{B}}_0$, is only valid if the model $\mathbf{f}\left(\mathsf{\theta }\right)$ (and hence the likelihood function, $L_n\left(\mathsf{\theta }\right)$) is correctly specified [7]. If the model is misspecified, the sensitivity and variability matrices are misaligned [8] and asymptotic $100(1-\alpha )$% credible intervals will usually have less than nominal frequentist coverage probabilities. Thus, Bayesian credible sets of confidence level $\gamma =100(1-\alpha )$% cannot be interpreted as confidence sets of level $\gamma$% [9]. This so-called overconditioning [10,11,12] is a result of the customary aleatoric treatment of residuals when, in fact, they are nonrandom (systematic) in nature and manifest with an unduly small parameter uncertainty and poorly calibrated prediction intervals [13,14,15,16,17]. In such cases, interpretation of the posterior parameter distribution $P\left(\mathsf{\theta }\right)L_n\left(\mathsf{\theta }\right)$ may be problematic. Not only can the posterior $P_n\left(\mathsf{\theta }\right)$ fail to provide a valid probabilistic description of information about $\mathsf{\theta }$, but it may also be unclear whether $\mathsf{\theta }$ corresponds to any meaningful or scientifically relevant quantities [18].

Upon misspecification, the true parameter values ${\mathsf{\theta }}_0$ of the data generating process are not in the model parameter space $\mathsf{\theta }\in \mathsf{\Theta }\subseteq {\mathbb{R}}^d$ (see Figure 1).

The best attainable values of the parameters or so-called pseudo-true parameter values ${\mathsf{\theta }}_{*}$ minimize the Kullback and Leibler [19] divergence between the true probability density function $q_{\mathsf{\Omega }}(\omega \mid {\mathsf{\theta }}_0)$ of $\mathsf{\Omega }$ and the incorrect family of densities $f(\omega \mid \mathsf{\theta })$ defined by $\mathsf{\theta }\in \mathsf{\Theta }$[8]. The consequences of misspecification are that the posterior distribution will now center on ${\mathsf{\theta }}_{*}$, the best distribution out of all distributions in the misspecified parametric family. However, a more pertinent problem is that the information identity ${\mathbf{A}}_{*}={\mathbf{B}}_{*}$ will not hold under misspecification. The ML estimator will still be asymptotically normal but now around the pseudo-true parameter values ${\widehat{\mathsf{\theta }}}_{*}$

$$\begin{array}{cc}\hfill & \sqrt{n}\left({\widehat{\mathsf{\theta }}}_{*}\right)\stackrel{\mathrm{d}}{⟶}{\mathcal{N}}_d\left(\mathbf{0},{\mathcal{G}}_0^{-1}\left({\mathsf{\theta }}_{*}\right)\right),\hfill \end{array}$$

where the covariance matrix of the estimator is ${\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{-1}{\mathbf{B}}_{*}^{\phantom{-1}}{\mathbf{A}}_{*}^{-1}={\displaystyle \frac{1}{n}}{\mathcal{G}}_0^{-1}\left({\mathsf{\theta }}_{*}\right)$ and ${\mathcal{G}}_0={\mathbf{A}}_{*}^{\phantom{-1}}{\mathbf{B}}_{*}^{-1}{\mathbf{A}}_{*}^{\phantom{-1}}$ is the so-called Godambe [20] information matrix. Thus, Godambe information ${\mathcal{G}}_0$ is the only valid currency of data information under misspecification. This information matrix guarantees asymptotically valid parameter confidence intervals and standard errors even when the likelihood function $L_n\left(\mathsf{\theta }\right)$ is incorrectly specified [21].

This paper builds on Vrugt et al. [8] and addresses the fundamental limitation that Bayesian methods do not provide asymptotically valid standard errors when the model is misspecified [22,23,24,25]. The asymptotic covariance matrix of Markov chain Monte Carlo (MCMC) simulation methods is the inverse of a single “slice of bread,” ${\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{-1}$, rather than the asymptotically valid sandwich matrix ${\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{-1}{\mathbf{B}}_{*}^{\phantom{-1}}{\mathbf{A}}_{*}^{-1}$. Analytic and numerical case studies in Vrugt et al. [8] confirm that the posterior distribution can significantly overestimate the informativeness of streamflow measurements, resulting in an overly optimistic model and parameter uncertainty estimates under misspecification. The sandwich estimator, by contrast, substantially widens the credible intervals for watershed model parameters and discharge. This theoretical inconsistency between Bayesian and frequentist approaches warrants a closer look at MCMC methodology, specifically, how we might adapt the Metropolis–Hastings (MH) algorithm [26,27] so that the stationary distribution of the Markov chains reflects the correct sandwich asymptotics. The general problem we address is that Bayesian methods yield $\mathsf{\theta }\sim {\mathcal{N}}_d({\widehat{\mathsf{\theta }}}_{*},{\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{-1})$ as the asymptotic description of the posterior parameter distribution $P_n\left(\mathsf{\theta }\right)=P\left(\mathsf{\theta }\right)L_n\left(\mathsf{\theta }\right)$ or $P_n\left(\mathsf{\theta }\right)\propto exp\left({\mathcal{P}}_n\left(\mathsf{\theta }\right)\right)$, whereas this should be $\mathsf{\theta }\sim {\mathcal{N}}_d({\widehat{\mathsf{\theta }}}_{*},{\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{-1}{\mathbf{B}}_{*}^{\phantom{-1}}{\mathbf{A}}_{*}^{-1})$ when the model is misspecified. This reconciliation of frequentist asymptotic theory with Bayesian computational procedures is of great practical importance, particularly for applications that make use of prior information, latent variables, and/or hierarchical models. We wish to enhance the robustness of Bayesian computation under model misspecification, while retaining the flexibility and coherence of MCMC simulation methods. We view this not as an attempt to force Bayesian and frequentist methods to align, but as a practical safeguard in applications where model assumptions are inevitably imperfect. Moreover, the strictly proper scoring rules we propose as a byproduct of the misalignment between the sensitivity and variability matrices offer information-theoretically principled metrics for quantifying the degree of misspecification and for guiding model selection and improvement.

The goals of this paper are three-fold. First, we review and examine existing methods for obtaining an asymptotically valid description of the sandwich posterior distribution using MCMC sampling methods. Then, as second objective, we introduce a new and more rigorous sandwich sampling method which overcomes limitations of currently available methods. In particular, existing approaches often rely on a single scalar correction factor applied uniformly across all parameters, which can fail to capture directional asymmetries in the sandwich distribution especially under model misspecification or for small-sample sizes. Our proposed method addresses this limitation by introducing a direction-dependent scaling factor or learning rate that adapts to the local curvature of the sandwich distribution. As third and last objective of this paper, we present an information-theoretic interpretation of the strictly proper alignment score proposed by Vrugt et al. [8], which quantifies the concordance between matrices ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$. Several other scalar indicators of model misspecification are also introduced in this section.

The theory and methodology of this paper are an integral part of DREAM-Suite, a Matlab-Python software package for Bayesian model training, evaluation and diagnostics [28]. This software can be downloaded from the first author’s GitHub account https://github.com/jaspervrugt (accessed on 3 September 2025) and includes the different case studies presented herein.

2. Notation and Definitions

Boldface uppercase letters denote matrices, $\mathbf{A}$, boldface lowercase letters signify vectors, $\mathbf{a}$, and italic lowercase letters are used for scalars, a. The superscripts “⊤” and “$-1$” stand for matrix transpose and matrix inverse, respectively. By default, we assume column vectors and, thus, we write $\mathbf{a}={(a_1,\dots ,a_n)}^{\top }$ for a $n\times 1$ vector. If $\mathbf{X}={(X_1,\dots ,X_d)}^{\top }$ is a vector of d random variables, then we say that its expectation is the vector $\mathsf{\mu }={({\mu }_1,\dots ,{\mu }_d)}^{\top }$ and write $\mathsf{\mu }=\mathbb{E}\left[\mathbf{X}\right]$, thus combining d scalar equations into one vector equation. The variance of random vector $\mathbf{X}$ is the $d\times d$ matrix $\mathsf{\Sigma }$ whose $(i,j)$th element is

$$\begin{array}{cc}\hfill Cov[X_i,X_j]& =\mathbb{E}\left[(X_i-{\mu }_i)(X_j-{\mu }_j)\right],\hfill \end{array}$$

where $i,j\in (1,\dots ,d)$. In vector notation, we write

$$\begin{array}{cc}\hfill Var\left[\mathbf{X}\right]& =\mathbb{E}\left[(\mathbf{X}-\mathsf{\mu }){(\mathbf{X}-\mathsf{\mu })}^{\top }\right],\hfill \end{array}$$

thus combining $d^2$ scalar equations into one matrix equation. In this formulation, $\mathbf{X}-\mathsf{\mu }$, is a $d\times 1$ vector and the outer (cross) product of $\mathbf{X}-\mathsf{\mu }$ and ${(\mathbf{X}-\mathsf{\mu })}^{\top }$ returns a $d\times d$ matrix. The inner or dot product of two n-vectors $\mathbf{a}$ and $\mathbf{b}$ is equal to ${\mathbf{a}}^{\top }\mathbf{b}$ and returns a scalar. For notational convenience, we write ${\mathbf{Z}}_n^z\left(\mathsf{\theta }\right)$ instead of ${\left({\mathbf{Z}}_n\left(\mathsf{\theta }\right)\right)}^z$, where the superscript $z\in \{-1,\top \}$ denotes either matrix inversion or transposition, respectively. This convention applies to any matrix $\mathbf{Z}$, such as $\mathbf{A}$, $\mathbf{B}$, $\mathcal{I}$, and $\mathcal{G}$. In the same spirit, we write ${\nabla }_{\mathsf{\theta }}^{\top }{\mathcal{L}}_{\omega }\left(\mathsf{\theta }\right)$ to denote the transpose of the gradient vector ${\nabla }_{\mathsf{\theta }}{\mathcal{L}}_{\omega }\left(\mathsf{\theta }\right)$, so that outer products are written compactly.

Suppose ${\mathsf{\theta }}_{*}$ are the pseudo-true parameter values of the data-generating process $\mathfrak{S}$ and $\mathsf{\omega }={({\omega }_1,\dots ,{\omega }_n)}^{\top }$ and $\mathbf{y}={(y_1,\dots ,y_n)}^{\top }$ are $n\times 1$ vectors of materialized and modeled outcomes, respectively. Then, the most important scalars, vectors, and matrices are

• The likelihood is a scalar and denoted $L_{\omega }\left(\mathsf{\theta }\right)$ for a single datum $\omega$. For a data set ${\omega }_1,\dots ,{\omega }_n$, we write $L_n\left(\mathsf{\theta }\right)$. The symbol ${\mathcal{L}}_n\left(\mathsf{\theta }\right)$ denotes the natural logarithm of $L_n\left(\mathsf{\theta }\right)$.
• The $d\times d$ Hessian matrix ${\mathbf{H}}_{\omega }\left(\mathsf{\theta }\right)={\nabla }^2{\mathcal{L}}_{\omega }\left(\mathsf{\theta }\right)$ contains the second-order partial derivatives of ${\mathcal{L}}_{\omega }\left(\mathsf{\theta }\right)$ w.r.t. $\mathsf{\theta }$. The total Hessian is given by ${\mathbf{H}}_n\left(\mathsf{\theta }\right)={\sum }_{i=1}^n{\mathbf{H}}_{{\omega }_i}\left(\mathsf{\theta }\right)$, equivalently ${\mathbf{H}}_n\left(\mathsf{\theta }\right)={\nabla }^2{\mathcal{L}}_n\left(\mathsf{\theta }\right)$.
• The $d\times d$ sensitivity matrix is defined as ${\mathbf{A}}_n=-{\displaystyle \frac{1}{n}}{\mathbf{H}}_n\left({\widehat{\mathsf{\theta }}}_{*}\right)$ with probability limit ${\mathbf{A}}_{*}=plim{\mathbf{A}}_n$.
• The $d\times d$ variability matrix is defined as ${\mathbf{B}}_n={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n\nabla {\mathcal{L}}_{{\omega }_i}\left({\widehat{\mathsf{\theta }}}_{*}\right){\nabla }^{\top }{\mathcal{L}}_{{\omega }_i}\left({\widehat{\mathsf{\theta }}}_{*}\right)$ with probability limit ${\mathbf{B}}_{*}=plim{\mathbf{B}}_n$.
• The $d\times d$ Fisher information matrix ${\mathcal{I}}_n\left({\mathsf{\theta }}_{*}\right)={\mathbb{E}}_{\omega }[\nabla {\mathcal{L}}_n\left({\mathsf{\theta }}_{*}\right){\nabla }^{\top }{\mathcal{L}}_n\left({\mathsf{\theta }}_{*}\right)]$ is the expectation w.r.t. $\omega$ of the outer product of the gradient of the log-likelihood evaluated at ${\mathsf{\theta }}_{*}$.
• The matrix inverse of the Fisher information ${\mathcal{I}}_n^{-1}\left({\mathsf{\theta }}_{*}\right)$ is a $d\times d$ covariance matrix. Under correct specification this naive variance equals the asymptotic variance of the ML estimator.
• The $d\times d$ Godambe information matrix is defined as ${\mathcal{G}}_n\left({\widehat{\mathsf{\theta }}}_{*}\right)=n{\mathbf{A}}_n^{\phantom{-1}}{\mathbf{B}}_n^{-1}{\mathbf{A}}_n^{\phantom{-1}}$, with probability limit ${\mathcal{G}}_0=plim{\displaystyle \frac{1}{n}}{\mathcal{G}}_n\left({\mathsf{\theta }}_{*}\right)={\mathbf{A}}_{*}^{\phantom{-1}}{\mathbf{B}}_{*}^{-1}{\mathbf{A}}_{*}^{\phantom{-1}}$.
• The matrix inverse of the Godambe information ${\mathcal{G}}_n^{-1}\left({\widehat{\mathsf{\theta }}}_{*}\right)$ is a $d\times d$ covariance matrix. This robust or “sandwich” variance is a consistent estimator of the asymptotic variance of the ML estimator under misspecification.

Note that we omitted the subscript $\mathsf{\theta }$ in the vector differential operator ∇ as differentiation of the log-likelihood function is always with respect to the parameters.

The entries of the $d\times d$ “information” matrices ${\mathcal{I}}_n$, ${\mathbf{H}}_n$, and ${\mathcal{G}}_n$ grow linearly (on average) with n reflecting a steadily increasing amount of information about the unknown parameters $\mathsf{\theta }$ with more data. In contrast, ${\mathbf{A}}_n$ and ${\mathbf{B}}_n$ are sample averages of the sensitivity and variability matrices for n data points. Cameron and Trivedi [29] treat these two $d\times d$ matrices as estimators of ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$, respectively, the probability limits under the pseudo-true parameters ${\mathsf{\theta }}_{*}$. For the time being, we formulate all our information matrices ${\mathbf{A}}_{*}$, ${\mathbf{B}}_{*}$, ${\mathcal{I}}_n\left({\mathsf{\theta }}_{*}\right)$, and ${\mathcal{G}}_n\left({\mathsf{\theta }}_{*}\right)$ as population quantities as if the pseudo-true parameter values ${\mathsf{\theta }}_{*}$ of the data generating process are exactly known. In practice, the “information” matrices ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$ are replaced with empirical estimates ${\mathbf{A}}_n$ and ${\mathbf{B}}_n$, evaluated at the estimator ${\widehat{\mathsf{\theta }}}_{*}$ obtained from the ${\omega }_1,\dots ,{\omega }_n$. Further details are provided later.

Statistical distributions are designated as common symbols. If $\mathbf{X}$ is multivariate normally distributed with mean $\mathsf{\mu }\in {\mathbb{R}}^d$ and $d\times d$ covariance matrix $\mathsf{\Sigma }=Var\left[\mathbf{X}\right]$, we write $\mathbf{X}\sim {\mathcal{N}}_d(\mathsf{\mu },\mathsf{\Sigma })$ and use $\mathbf{X}\sim {\mathcal{U}}_d(\mathbf{a},\mathbf{b})$ for the continuous d-variate uniform distribution on the closed-region $[\mathbf{a},\mathbf{b}]$, where $\mathbf{a},\mathbf{b}\in {\mathbb{R}}^{d\times 1}$ and $a_j<b_j$ for all $j=(1,\dots ,d)$. We write $P(\mathbf{X}\mid \mathsf{\omega })$ for the conditional pdf of $\mathbf{X}$ given the n-materialized outcomes $\mathsf{\omega }$. The Greek letter $\alpha \in (0,1)$ denotes the probability of rejecting the null hypothesis when the null hypothesis is true. We write $\gamma =1-\alpha$ for the confidence level.

3. Illustrative Example

Before discussing how to remedy Bayesian MCMC methods into sampling the asymptotically correct sandwich distribution, we first demonstrate the information identity ${\mathbf{A}}_{*}={\mathbf{B}}_{*}$ and failure thereof for a simple parametric model and synthetic data.

We revisit the first study of Vrugt et al. [8] and consider as data generating process $\mathsf{\Omega }\sim \mathcal{N}(\mu ,{\sigma }^2)$ of random variable $\mathsf{\Omega }$. We draw measurements ${\omega }_1,\dots ,{\omega }_n$ for $\mu =0$, ${\sigma }^2=1$ and $n=100$. As our model we use $y_i\sim N(m,s^2)$ with m unknown and $s^2$ fixed at some predefined value. If $s^2={\sigma }^2$, then the model is correctly specified, otherwise for $s^2\ne {\sigma }^2$ the model is misspecified. Now, we wish to determine the value of m using training data ${\omega }_1,\dots ,{\omega }_{100}$. The normal log-likelihood ${\mathcal{L}}_n^{\mathrm{n}}(m\mid s^2)$ is equal to

$$\begin{array}{cc}\hfill & {\mathcal{L}}_n^{\mathrm{n}}(m\mid s^2)=log\left(L_n^{\mathrm{n}}(m\mid s^2)\right)=-{\displaystyle \frac{n}{2}}log\left(2\pi s^2\right)-{\displaystyle \frac{1}{2}}{s}^{-2}\sum _{i=1}^n{({\omega }_i-m)}^2.\hfill \end{array}$$

(5)

Figure 2 displays ${\mathcal{L}}_n^{\mathrm{n}}(m\mid s^2)$ for $-5\le m\le 5$ using $s^2=1/2$ (red), $s^2=1$ (green) and $s^2=2$ (blue).

For $s^2=1$ (green line), the model is correctly specified and the information identity, ${\mathbf{A}}_{*}={\mathbf{B}}_{*}$ will hold. This implies that the expected value of the second derivative ${\ddot{\mathcal{L}}}_n^{\mathrm{n}}(m\mid s^2)$ of the log-likelihood function ${\mathcal{L}}_n^{\mathrm{n}}(m\mid s^2)$ at the likelihood maximum $\widehat{m}\approx \mu$ will equal the expected value of the squared first-derivative ${\dot{\mathcal{L}}}_n^{\mathrm{n}}(m\mid s^2)$ of ${\mathcal{L}}_n^{\mathrm{n}}(m\mid s^2)$ at this maximum, where the expectation is with respect to $\omega \in \mathsf{\Omega }$. This is easy to demonstrate with an analytic proof. The first and second derivatives of ${\mathcal{L}}_{\omega }^{\mathrm{n}}(m\mid s^2)$ with respect to m are

$$\begin{array}{cccc}& \hfill & \hfill {\dot{\mathcal{L}}}_{\omega }^{\mathrm{n}}(m\mid s^2)& ={\displaystyle \frac{\mathrm{d}}{\mathrm{d}m}}{\mathcal{L}}_{\omega }^{\mathrm{n}}(m\mid s^2)=s^{-2}(\omega -m)\hfill \\ & \hfill & \hfill {\ddot{\mathcal{L}}}_{\omega }^{\mathrm{n}}(m\mid s^2)& ={\displaystyle \frac{{\mathrm{d}}^2}{\mathrm{d}{m}^2}}{\mathcal{L}}_{\omega }^{\mathrm{n}}(m\mid s^2)=-s^{-2}.\hfill \end{array}$$

(6)

The sensitivity matrix (a scalar in this case) is now equal to

$$\begin{array}{cc}\hfill & {\mathbf{A}}_{*}={\mathbb{E}}_{\omega }[-{\ddot{\mathcal{L}}}_{\omega }^{\mathrm{n}}(m\mid s^2)]={\mathbb{E}}_{\omega }\left[s^{-2}\right]=s^{-2},\hfill \end{array}$$

(7)

and the variability matrix (also a scalar here) at the likelihood maximum $m=\mu$ is

$$\begin{array}{cc}\hfill & {\mathbf{B}}_{*}={\mathbb{E}}_{\omega }\left[{\dot{\mathcal{L}}}_{\omega }^{\mathrm{n}}(m\mid s^2){\dot{\mathcal{L}}}_{\omega }^{\mathrm{n}}{(m\mid s^2)}^{\top }\right]={\mathbb{E}}_{\omega }\left[s^{-2}(\omega -m)s^{-2}(\omega -m)\right]=s^{-4}{\sigma }^2.\hfill \end{array}$$

(8)

If we assume the variance $s^2={\sigma }^2$ of the green line in Figure 2, then ${\mathbf{B}}_{*}=s^{-2}$. This is equal to the sensitivity matrix ${\mathbf{A}}_{*}=s^{-2}$ from Equation (7), thus ${\mathbf{A}}_{*}={\mathbf{B}}_{*}$. In this correctly specified case and with variance known, an exact $100(1-\alpha )\%$ confidence interval for $\widehat{m}$ is

$$\widehat{m}\pm {\mathsf{\Phi }}^{-1}(1-\frac{1}{2}\alpha )\sqrt{{\sigma }^2/n}.$$

where ${\mathsf{\Phi }}^{-1}\left(p_{\alpha }\right)$ is the quantile function of the standard normal distribution evaluated at percentile $p_{\alpha }=1-{\displaystyle \frac{1}{2}}\alpha$. This confidence interval for $\widehat{m}$ coincides with the classical frequentist interval estimate of the sample mean [30].

For the other two models with $s^2=1/2$ (red line) and $s^2=2$ (blue line) ${\mathbf{A}}_{*}\ne {\mathbf{B}}_{*}$, and consequently the naive variance ${\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{-1}=s^2/n$ will underestimate and overestimate, respectively, the actual uncertainty of m. Upon misspecification, the sandwich variance ${\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{-1}{\mathbf{B}}_{*}^{\phantom{-1}}{\mathbf{A}}_{*}^{-1}=s^2{s}^{-4}{\sigma }^2{s}^2/n={\sigma }^2/n$ equals the correct variance (${\sigma }^2/n)$ of m. This estimator does not require prior knowledge of ${\sigma }^2$ as the matrices ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$ are replaced by their sample estimates, ${\mathbf{A}}_n$ and ${\mathbf{B}}_n$, evaluated at $\mathsf{\theta }={\widehat{\mathsf{\theta }}}_{*}$.

In Appendix A, we verify that the variability matrix satisfies the variance rule given in Equation (4), confirming that ${\mathbf{B}}_{*}=Var\left[{\dot{\mathcal{L}}}_{\omega }^{\mathrm{n}}(m\mid s^2)\right]$.

4. Sandwich-Adjusted MCMC Simulation

4.1. The Metropolis–Hastings Algorithm

We must summarize the posterior parameter distribution, $P_n\left(\mathsf{\theta }\right)\propto P\left(\mathsf{\theta }\right)L_n\left(\mathsf{\theta }\right)$. When this task cannot be carried out by analytical means nor by analytical approximation, Monte Carlo simulation methods can be used to generate samples from the posterior distribution.

The basis of MCMC simulation is a Markov chain that generates a random walk through the search space and successively visits solutions with stable frequencies stemming from a stationary distribution, $P_n\left(\mathsf{\theta }\right)$. Assume that the points $\{{\mathsf{\theta }}_{\left(0\right)},{\mathsf{\theta }}_{\left(1\right)},\dots ,{\mathsf{\theta }}_{(t-1)}\}$ have already been sampled, then the MH algorithm [26,27] proceeds as follows (see Algorithm 1). At iteration t, the transition kernel $q({\mathsf{\theta }}_{\mathrm{p}}\mid {\mathsf{\theta }}_{(t-1)})$ generates a trial move ${\mathsf{\theta }}_{\mathrm{p}}$ around the current chain state ${\mathsf{\theta }}_{(t-1)}$. Next, this candidate point is accepted with MH probability

$$P_{\mathrm{acc}}({\mathsf{\theta }}_{(t-1)}\to {\mathsf{\theta }}_{\mathrm{p}})=min\left\{1,{\displaystyle \frac{P\left({\mathsf{\theta }}_{\mathrm{p}}\right)L_n\left({\mathsf{\theta }}_{\mathrm{p}}\right)q({\mathsf{\theta }}_{(t-1)}\mid {\mathsf{\theta }}_{\mathrm{p}})}{P\left({\mathsf{\theta }}_{(t-1)}\right)L_n\left({\mathsf{\theta }}_{(t-1)}\right)q({\mathsf{\theta }}_{\mathrm{p}}\mid {\mathsf{\theta }}_{(t-1)})}}\right\},$$

(9)

and we set ${\mathsf{\theta }}_{\left(t\right)}={\mathsf{\theta }}_{\mathrm{p}}$, otherwise if the candidate point is rejected, the chain remains at is old position, ${\mathsf{\theta }}_{\left(t\right)}={\mathsf{\theta }}_{(t-1)}$. Repeated application of these steps results in a Markov chain $\{{\mathsf{\theta }}_{\left(0\right)},{\mathsf{\theta }}_{\left(1\right)},\dots ,{\mathsf{\theta }}_{\left(T\right)}\}$ which, under certain regularity conditions, has a unique stationary distribution with posterior probability density function, $P_n\left(\mathsf{\theta }\right)$. In practice, this means that if one looks at the values of $\mathsf{\theta }$ sufficiently far from the arbitrary initial value, ${\mathsf{\theta }}_{\left(0\right)}$, the successively generated states of the chain will be distributed according to $P_n\left(\mathsf{\theta }\right)$, the posterior probability distribution of $\mathsf{\theta }$. This so-called burn-in period $\{{\mathsf{\theta }}_{\left(0\right)},{\mathsf{\theta }}_{\left(1\right)},\dots ,{\mathsf{\theta }}_{(b-1)}\}$, where $b\ll T$ is required to allow the chain to travel to the high-probability density (HPD) region of the target distribution. Thus, the last $M=T-b+1$ samples $\{{\mathsf{\theta }}_{\left(b\right)},{\mathsf{\theta }}_{(b+1)},\dots ,{\mathsf{\theta }}_{\left(T\right)}\}$ are used to approximate the posterior parameter distribution, $P_n\left(\mathsf{\theta }\right)\propto P\left(\mathsf{\theta }\right)L_n\left(\mathsf{\theta }\right)$.

Algorithm 1 Metropolis–Hastings (MH)

Input: Prior, $P\left(\mathsf{\theta }\right)$, likelihood, $L_n\left(\mathsf{\theta }\right)$, and transition density, $q({\mathsf{\theta }}_{\mathrm{p}}\mid {\mathsf{\theta }}_{(t-1)})$         Total number of samples T Output: Samples $\{{\mathsf{\theta }}_{\left(0\right)},{\mathsf{\theta }}_{\left(1\right)},\dots ,{\mathsf{\theta }}_{\left(T\right)}\}$ with stationary distribution $P_n\left(\mathsf{\theta }\right)\propto P\left(\mathsf{\theta }\right)L_n\left(\mathsf{\theta }\right)$ Draw initial chain state ${\mathsf{\theta }}_{\left(0\right)}$ from the prior distribution, ${\mathsf{\theta }}_{\left(0\right)}\sim P\left(\mathsf{\theta }\right)$ for  $t=1$  to  T  do     Sample a proposal, ${\mathsf{\theta }}_{\mathrm{p}}\sim q(·\mid {\mathsf{\theta }}_{(t-1)})$, from the transition kernel     Compute the acceptance probability $P_{\mathrm{acc}}({\mathsf{\theta }}_{(t-1)}\to {\mathsf{\theta }}_{\mathrm{p}})$ using Equation (9)     if $P_{\mathrm{acc}}({\mathsf{\theta }}_{(t-1)}\to {\mathsf{\theta }}_{\mathrm{p}})\ge Z$ then         Accept the candidate point, ${\mathsf{\theta }}_{\left(t\right)}={\mathsf{\theta }}_{\mathrm{p}}$ and $P_n\left({\mathsf{\theta }}_{\left(t\right)}\right)=P_n\left({\mathsf{\theta }}_{\mathrm{p}}\right)$     else         Reject the proposal and set ${\mathsf{\theta }}_{\left(t\right)}={\mathsf{\theta }}_{(t-1)}$ and $P_n\left({\mathsf{\theta }}_{\left(t\right)}\right)=P_n\left({\mathsf{\theta }}_{(t-1)}\right)$     end if end for Return:  $\{{\mathsf{\theta }}_{\left(0\right)},{\mathsf{\theta }}_{\left(1\right)},\dots ,{\mathsf{\theta }}_{\left(T\right)}\}$

In the limit of $T\to \infty$, the MAP density estimate ${\widehat{\mathsf{\theta }}}_n$ of the sampled Markov chain will converge to the true parameter values ${\mathsf{\theta }}_0$ of the data generating process

$$\begin{array}{cc}\hfill & \underset{n,T\to \infty }{lim}{\widehat{\mathsf{\theta }}}_n={\mathsf{\theta }}_0,\hfill \end{array}$$

or under misspecification, we write ${lim}_{n,T\to \infty }{\widehat{\mathsf{\theta }}}_n={\mathsf{\theta }}_{*}$. The asymptotic covariance matrix of the Markov chain will equal the matrix inverse of a single “slice of bread”

$$\begin{array}{cc}\hfill & \underset{n,T\to \infty }{lim}nCov\left[\{{\mathsf{\theta }}_{\left(b\right)},{\mathsf{\theta }}_{(b+1)},\dots ,{\mathsf{\theta }}_{\left(T\right)}\}\right]={\mathbf{A}}_{*}^{-1}.\hfill \end{array}$$

Thus, after burn-in, the covariance matrix of the chain draws is the nonlinear sample equivalent of ${\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{-1}$, whereas we desire this to be

$$\begin{array}{cc}\hfill & \underset{n,T\to \infty }{lim}nCov\left[\{{\mathsf{\theta }}_{\left(b\right)},{\mathsf{\theta }}_{(b+1)},\dots ,{\mathsf{\theta }}_{\left(T\right)}\}\right]={\mathbf{A}}_{*}^{-1}{\mathbf{B}}_{*}^{\phantom{-1}}{\mathbf{A}}_{*}^{-1}.\hfill \end{array}$$

In the language of Shaby [25], we want to complete the sandwich by joining the slice of bread ${\mathbf{A}}_{*}^{-1}$ to the open-faced sandwich ${\mathbf{B}}_{*}^{\phantom{-1}}{\mathbf{A}}_{*}^{-1}$ to obtain the desired sandwich covariance.

It would be ideal if we could reformulate the recipe of Algorithm 1 so that the sampled Markov chains always converge to the right asymptotic distribution, which under misspecification is the sandwich estimator. This has proven to be a formidable task. The culprit is the implicit assumption in Bayes law that the model correctly describes the relationship between prior, likelihood, and evidence. We can relax this assumption with the use of so-called belief distributions, but it is not immediately clear how to turn this new paradigm into an MH recipe with correct limiting distribution under misspecification.

We do not delve into the MCMC theory, but rather focus our attention on more practical remedies that help adjust the random walk of Algorithm 1 to the sandwich distribution. Existing methods for doing so transform either the likelihood function $L_n\left(\mathsf{\theta }\right)$ or the parameter values $\mathsf{\theta }$ to match the curvature of the sandwich distribution around ${\mathsf{\theta }}_{*}$. All these methods assume knowledge of the MAP parameter values ${\widehat{\mathsf{\theta }}}_{*}$ and the sensitivity ${\mathbf{A}}_{*}$ and variability ${\mathbf{B}}_{*}$ matrices at $\mathsf{\theta }={\mathsf{\theta }}_{*}$. Next, we review two existing recipes based on magnitude- and curvature-adjustments of the log-likelihood function to sample the posterior sandwich distribution. Then, we present the theory of a third, more rigorous, and convenient approach, which we coin the kernel-amendment method for sandwich-adjusted MCMC simulation.

4.2. Algorithm 1: Magnitude Adjustment

We can adjust the magnitude of the log-likelihood function to enforce the sandwich variance matrix ${\mathsf{\Sigma }}_n^{\mathrm{sand}}$ on the posterior realizations of the sampled Markov chain(s). If the log-likelihood function ${\mathcal{L}}_n\left(\mathsf{\theta }\right)$ is approximately quadratic in a neighborhood of the maximum a posteriori (MAP) parameter values ${\widehat{\mathsf{\theta }}}_{*}$, then posterior exploration of the scaled log-likelihood $k{\mathcal{L}}_n\left(\mathsf{\theta }\right)$ via the MH algorithm should yield a good approximation to the sandwich-adjusted posterior. This so-called omnibus adjustment was originally proposed by Pauli et al. [31] as a correction to the second Bartlett identity, and the scalar k is estimated following the procedures outlined by Ribatet et al. [24] and di San Miniato and Sartori [32]

$$k={\displaystyle \frac{d}{tr\left({\left({\mathsf{\Sigma }}_n^{\mathrm{naive}}\right)}^{-1}{\mathsf{\Sigma }}_n^{\mathrm{sand}}\right)}}\stackrel{\mathrm{p}}{⟶}{\displaystyle \frac{d}{tr\left({\mathbf{A}}_{*}^{-1}{\mathbf{B}}_{*}^{\phantom{-1}}\right)}}.$$

(10)

The unary trace operator $tr\left(·\right)$ returns the sum of the diagonal elements of the $d\times d$ matrix ${\mathbf{A}}_{*}^{-1}{\mathbf{B}}_{*}^{\phantom{-1}}$. This trace is equal to the sum of the eigenvalues of the matrix-matrix product ${\mathbf{A}}_{*}^{-1}{\mathbf{B}}_{*}^{\phantom{-1}}$. The omnibus adjustment can be thought of as a tempering of the log-likelihood function and flattens ${\mathcal{L}}_n\left(\mathsf{\theta }\right)$ for $0<k<1$, thereby slowing down learning and matching the informativeness of the data with the Godambe information ${\mathcal{G}}_n\left({\mathsf{\theta }}_{*}\right)=n{\mathbf{A}}_{*}^{\phantom{-1}}{\mathbf{B}}_{*}^{-1}{\mathbf{A}}_{*}^{\phantom{-1}}$. The MH algorithm with product of the omnibus scalar k and log-likelihood ${\mathcal{L}}_n\left(\mathsf{\theta }\right)$, will from now on be referred to as Algorithm 1.

Remark 1. 

The omnibus scalar k is reminiscent of a learning rate in a power likelihood function, where ’learning rate’ refers to the extent to which the data influence posterior updating. A value of $k<1$ downweights the data and reduces sensitivity to outliers, while $k>1$ increases the influence of the data on the posterior distribution. To provide a deeper intuition of the advantages of a power likelihood, we revisit example 1 in Section 3. Suppose that we inadvertently assumed that $s^2=2$ (blue line in Figure 2) and the normal distribution model is misspecified. Our estimate of the (naive) variance of $\widehat{m}$ will be ${\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{-1}=s^2/n=2/n$, while the true variance of $\widehat{m}$ is ${\sigma }^2/n$ or $1/n$. With the omnibus scalar, the sensitivity and variability matrices of $k{\mathcal{L}}_n^{\mathrm{n}}(m\mid s^2)$ are equal to ${\mathbf{A}}_{*}=k{s}^{-2}$ and ${\mathbf{B}}_{*}=k^2{s}^{-4}{\sigma }^2$, respectively (see Appendix C). For $s^2=2$, we have ${\mathbf{A}}_{*}=k/2$ and ${\mathbf{B}}_{*}=k^2/4$. The information identity ${\mathbf{A}}_{*}={\mathbf{B}}_{*}$ holds if $k^2-2k=0$, yielding, $k=2$. At this value of k, the naive variance of $k{\mathcal{L}}_n^{\mathrm{n}}\left(\mathsf{\theta }\right)$ is ${\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{-1}=2k^{-1}/n$ or $1/n$, which is the correct variance of $\widehat{m}$ as ${\sigma }^2=1$. Thus, the idea behind the omnibus scalar k is to choose its value such that the powered likelihood $L_n^k\left(\mathsf{\theta }\right)$ satisfies the information identity ${\mathbf{A}}_{*}={\mathbf{B}}_{*}$. As a result, the naive posterior parameter distribution under $L_n^k\left(\mathsf{\theta }\right)$ coincides with the sandwich distribution of the original likelihood $L_n\left(\mathsf{\theta }\right)$. This omnibus correction yields the correct variance-covariance matrix for the estimator ${\widehat{\mathsf{\theta }}}_{*}$.

Remark 2. 

The use of a single scalar k for all d parameters may suffice when the sandwich distribution is well approximated by a multivariate Gaussian, that is, when the posterior is nearly symmetric and its surface is approximately quadratic around ${\widehat{\mathsf{\theta }}}_{*}$. However, in the presence of model misspecification, prior truncation, or directional heterogeneity in sensitivity, a global scalar k will distort the shape of the adjusted posterior distribution. In such cases, a separate scaling factor is required for each dimension to preserve the local geometry and asymmetry of the sandwich distribution. This directional asymmetry becomes especially pronounced in small-sample settings, say, $n<100$, where the central limit approximation does not hold and curvature varies across parameters.

Remark 3. 

Other definitions of the scalar k have been proposed in the literature [33], including those based on moment-matching conditions [34], adjustments to degrees of freedom [35] inspired by the Satterthwaite–Welch method [36,37], and alternative scaling approaches [38]. See also Varin et al. [39] for a broader overview.

Remark 4. 

The exponent in a power likelihood is typically denoted by λ rather than k.

4.3. Algorithm 2: Curvature Adjustment

While the asymptotic covariance of ${\widehat{\mathsf{\theta }}}_{*}$ is the sandwich matrix ${\mathsf{\Sigma }}_n^{\mathrm{sand}}$, the MH algorithm instead yields ${\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{-1}$, the inverse of a single “slice of bread”. In the words of Shaby [25], we wish to complete the sandwich by attaching this slice to the open-faced piece ${\mathbf{B}}_{*}^{\phantom{-1}}{\mathbf{A}}_{*}^{-1}$ to recover the full sandwich covariance ${\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{-1}{\mathbf{B}}_{*}^{\phantom{-1}}{\mathbf{A}}_{*}^{-1}$. We now review two approaches that adjust the curvature of the log-likelihood to match this target distribution.

4.3.1. A-Posteriori Adjustment

Let $\{{\mathsf{\theta }}_{\left(b\right)},{\mathsf{\theta }}_{(b+1)},\dots ,{\mathsf{\theta }}_{\left(T\right)}\}$ denote the post-burn-in samples from a Markov chain of length T, where the first b realizations are discarded. This yields $M=T-b+1$ samples drawn from the naive posterior $P_n\left(\mathsf{\theta }\right)\propto P\left(\mathsf{\theta }\right)L_n\left(\mathsf{\theta }\right)$ based on a data sample of size n. We center these samples at a reference point ${\widehat{\mathsf{\theta }}}_{*}$ (e.g., posterior mode or mean) and pre-multiply by a $d\times d$ matrix ${\mathsf{\Psi }}_{*}$ to obtain open-face sandwich (OFS)-adjusted samples [25]

$$\begin{array}{cc}\hfill {\mathsf{\theta }}_{\left(j\right)}^{\mathrm{ofs}}& ={\widehat{\mathsf{\theta }}}_{*}+{\mathsf{\Psi }}_{*}({\mathsf{\theta }}_{\left(j\right)}-{\widehat{\mathsf{\theta }}}_{*}),j=1,\dots ,M.\hfill \end{array}$$

(11)

This linear map applies direction-specific dilations along the principal axes of the local posterior ellipsoid without unnecessary rotation [38], thereby changing the local geometry of the naive posterior to that of the sandwich distribution. A convenient choice is [25]

$$\begin{array}{cc}\hfill {\mathsf{\Psi }}_{*}^{\phantom{\top }}& ={\mathbf{A}}_{*}^{-1\phantom{1/2}}{\mathbf{B}}_{*}^{1/2}{\mathbf{A}}_{*}^{1/2}.\hfill \end{array}$$

(12)

Under standard regularity conditions, the naive posterior is locally Gaussian with covariance ${\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{-1}$. After centering at ${\widehat{\mathsf{\theta }}}_{*}$ and applying the transformation ${\mathsf{\Psi }}_{*}$, we obtain

$$\begin{array}{cc}\hfill Cov\left({\mathsf{\theta }}^{\mathrm{ofs}}\right)=Cov\left({\mathsf{\Psi }}_{*}^{\phantom{\top }}(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*})\right)& ={\mathsf{\Psi }}_{*}^{\phantom{\top }}\left({\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{-1}\right){\mathsf{\Psi }}_{*}^{\top }\hfill \\ \hfill & ={\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{-1\phantom{1/2}}{\mathbf{B}}_{*}^{1/2}{\mathbf{A}}_{*}^{1/2}{\mathbf{A}}_{*}^{-1\phantom{1/2}}{\mathbf{A}}_{*}^{1/2}{\mathbf{B}}_{*}^{1/2}{\mathbf{A}}_{*}^{-1\phantom{1/2}}\hfill \\ \hfill & ={\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{-1}{\mathbf{B}}_{*}^{\phantom{-1}}{\mathbf{A}}_{*}^{-1},\hfill \end{array}$$

which is precisely the asymptotic sandwich covariance. Here, ${\mathbf{A}}_{*}^{1/2}$ is the principal square root of the symmetric positive definite matrix ${\mathbf{A}}_{*}$ such that ${\mathbf{A}}_{*}^{1/2}·{\mathbf{A}}_{*}^{1/2}={\mathbf{A}}_{*}$.

This a posteriori correction method is computationally appealing, as it does not require additional evaluations of the likelihood function $L_n\left(\mathsf{\theta }\right)$. However, the OFS-adjusted posterior samples may not accurately represent the true sandwich parameter distribution, particularly if the adjustment matrix ${\mathsf{\Psi }}_{*}$ is not constant over the region of ${\widehat{\mathsf{\theta }}}_{*}$ in which there is sandwich parameter uncertainty. However, this assumption is not unique to curvature-based methods. What is specific to such methods is the nonuniqueness of the matrix square roots ${\mathbf{A}}_{*}^{1/2}$ and ${\mathbf{B}}_{*}^{1/2}$ when the sensitivity and variability matrices ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$ are not positive semi-definite. This can occur if the quadratic approximation of ${\mathcal{L}}_n\left(\mathsf{\theta }\right)$ via the second-order Taylor expansion at $\mathsf{\theta }={\widehat{\mathsf{\theta }}}_{*}$ does not adequately describe the actual curvature of the log-likelihood function. In such cases, ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$ may not be symmetric. Moreover, under certain conditions, these matrices may be ill-conditioned or nearly singular. This may arise when two or more parameters exhibit strong linear dependence, when parameters have very different magnitudes, and/or when the sample size n is small. Ill-conditioning can also be introduced as an artifact of numerical approximation, particularly when finite differences are used to estimate the first- and second-order partial derivatives of ${\mathcal{L}}_n\left(\mathsf{\theta }\right)$ with respect to $\mathsf{\theta }$. Floating-point arithmetic can lead to numerical instability due to rounding errors and subtractive cancellation when evaluating small differences between almost equal numbers. This will distort the computation of ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$.

We can symmetrize a matrix $\mathbf{Z}$ by working with $\frac{1}{2}(\mathbf{Z}+{\mathbf{Z}}^{\top })$ instead. Ill-conditioning can be addressed through Tikhonov regularization by adding to ${\mathbf{A}}_{*}$ and/or ${\mathbf{B}}_{*}$ a diagonal matrix, $ϵ{\mathbf{I}}_d$, where ${\mathbf{I}}_d$ is the $d\times d$ identity matrix and $ϵ>0$ is a small positive scalar. This technique, also known as ridge regression, changes the eigenvalues of the matrix from ${\underline{\lambda }}_1,\dots ,{\underline{\lambda }}_d$ to ${\underline{\lambda }}_1+ϵ,\dots ,{\underline{\lambda }}_d+ϵ$. If all eigenvalues are positive and the matrix is symmetric, then its principal square roots will be unique. Throughout this paper, we assume that the matrices ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$ are positive definite.

The matrix square roots ${\mathbf{A}}_{*}^{1/2}$ and ${\mathbf{B}}_{*}^{1/2}$ of can be computed using different methods, including the generalized Cholesky factorization [40,41], singular value decomposition [42] or eigendecomposition [43,44]. These methods yield similar matrix square roots when ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$ are approximately symmetric and positive definite, since the key properties of symmetric positive definite matrices—such as real, positive eigenvalues and diagonalizability—still hold approximately in such cases. In this case, Cholesky factorization provides computationally inexpensive and stable estimates of ${\mathbf{A}}_{*}^{1/2}$ and ${\mathbf{B}}_{*}^{1/2}$[45]. However, if the matrices ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$ are far from symmetric, we should expect different matrix square roots, as the methods listed above have different optimality and invariance properties [46]. Preferably, the transformation from ${\mathsf{\theta }}_{\left(j\right)}$ to ${\mathsf{\theta }}_{\left(j\right)}^{\mathrm{ofs}}$ preserves directions of asymmetry. Singular value decomposition is numerically stable and preserves key geometric attributes of ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$ [25]. For symmetric matrices, the matrix square roots are given by

$$\begin{array}{cc}\hfill {\mathbf{A}}_{*}^{1/2}& ={\mathbf{U}}_a^{\phantom{1/2\top }}{\mathbf{D}}_a^{1/2}{\mathbf{U}}_a^{\phantom{1/2}\top }\mathrm{and}{\mathbf{B}}_{*}^{1/2}={\mathbf{U}}_b^{\phantom{1/2\top }}{\mathbf{D}}_b^{1/2}{\mathbf{U}}_b^{\phantom{1/2}\top },\hfill \end{array}$$

(13)

where ${\mathbf{U}}_a^{\phantom{\top }}{\mathbf{D}}_a^{\phantom{\top }}{\mathbf{U}}_a^{\top }$ and ${\mathbf{U}}_b^{\phantom{\top }}{\mathbf{D}}_b^{\phantom{\top }}{\mathbf{U}}_b^{\top }$ are the eigendecompositions of the sensitivity and variability matrices, respectively. Here, ${\mathbf{U}}_a$ and ${\mathbf{U}}_b$ are orthogonal matrices whose columns are the eigenvectors of ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$, and ${\mathbf{D}}^{1/2}=diag(\sqrt{{\underline{\lambda }}_1},\dots ,\sqrt{{\underline{\lambda }}_d})$, is the diagonal matrix of square roots of the corresponding eigenvalues.

The OFS adjustment of Equation (11) is equivalent to a linear transformation of the posterior samples. This transformation may not yield an accurate description of the true sandwich distribution if the model is highly nonlinear and/or the parameters are highly correlated. This difference between linear and nonlinear confidence intervals is well understood for the naive variance estimator [14,47,48,49,50], and these findings also apply to the sandwich estimator. In this situation, a priori adjustment (discussed next) will help determine whether the true sandwich confidence regions extend beyond this linear approximation. A consistent estimator of ${\mathsf{\Psi }}_{*}$ should generate credible intervals that are consistent 100$(1-\alpha )$% confidence intervals.

4.3.2. A Priori Adjustment

The OFS adjustment (Method 2a) transforms posterior samples post hoc, after the MH algorithm has completed sampling. While computationally appealing, this approach does not guarantee an accurate characterization of the true sandwich distribution under model misspecification. An alternative is to adjust the curvature of the likelihood function $L_n\left(\mathsf{\theta }\right)$ near the point estimate ${\widehat{\mathsf{\theta }}}_{*}$, thereby preserving the correct asymptotic behavior. Ribatet et al. [24] proposed achieving this by applying the affine transformation of Equation (11) during MCMC simulation

$$\begin{array}{cc}\hfill {\mathsf{\theta }}_{\mathrm{p}}^{\mathrm{ca}}& ={\widehat{\mathsf{\theta }}}_{*}+{\mathbf{C}}_{*}({\mathsf{\theta }}_{\mathrm{p}}-{\widehat{\mathsf{\theta }}}_{*}).\hfill \end{array}$$

(14)

The likelihood $L_n\left(\mathsf{\theta }\right)$ is then evaluated at the curvature-adjusted candidate point ${\mathsf{\theta }}_{\mathrm{p}}^{\mathrm{ca}}$ instead of ${\mathsf{\theta }}_{\mathrm{p}}$, resulting in the curvature-adjusted likelihood function

$$\begin{array}{cc}\hfill L_n^{\mathrm{ca}}\left(\mathsf{\theta }\right)& =L_n\left({\mathsf{\theta }}^{\mathrm{ca}}\right)=L_n\left({\widehat{\mathsf{\theta }}}_{*}+{\mathbf{C}}_{*}({\mathsf{\theta }}_{\mathrm{p}}-{\widehat{\mathsf{\theta }}}_{*})\right).\hfill \end{array}$$

(15)

The transformation in (14) does not change the location of the MAP solution. Indeed, if we enter ${\widehat{\mathsf{\theta }}}_{*}$ we yield that ${\mathsf{\theta }}_{\mathrm{p}}^{\mathrm{ca}}={\widehat{\mathsf{\theta }}}_{*}$, and, thus, the MAP solution is also a global maximum for $L_n^{\mathrm{ca}}\left(\mathsf{\theta }\right)$. The gradient or score and curvature (Hessian) of the curvature-adjusted log-likelihood ${\mathcal{L}}_n^{\mathrm{ca}}\left(\mathsf{\theta }\right)$ at $\mathsf{\theta }$ is equal to

$$\begin{array}{cc}\hfill \nabla {\mathcal{L}}_n^{\mathrm{ca}}\left(\mathsf{\theta }\right)& ={\mathbf{C}}_{*}^{\top }\nabla {\mathcal{L}}_n\left({\mathsf{\theta }}^{\mathrm{ca}}\right)\mathrm{and}{\nabla }^2{\mathcal{L}}_n^{\mathrm{ca}}\left(\mathsf{\theta }\right)={\mathbf{C}}_{*}^{\top }{\nabla }^2{\mathcal{L}}_n\left({\mathsf{\theta }}^{\mathrm{ca}}\right){\mathbf{C}}_{*}^{\phantom{\top }}.\hfill \end{array}$$

At the MAP estimator, the Hessian of the original log-likelihood ${\mathcal{L}}_n\left(\widehat{\mathsf{\theta }}\right)$ is asymptotically ${\nabla }^2{\mathcal{L}}_n\left({\widehat{\mathsf{\theta }}}_{*}\right)\approx -n{\mathbf{A}}_{*}$, so that

$$\begin{array}{c}\hfill {\nabla }^2{\mathcal{L}}_n^{\mathrm{ca}}\left({\widehat{\mathsf{\theta }}}_{*}\right)=-n{\mathbf{C}}_{*}^{\top }{\mathbf{A}}_{*}^{\phantom{\top }}{\mathbf{C}}_{*}^{\phantom{\top }}.\end{array}$$

To obtain the correct asymptotic curvature under model misspecification, we equate this expression to the sandwich information matrix

$$\begin{array}{cc}\hfill {\mathbf{C}}_{*}^{\top }{\mathbf{A}}_{*}^{\phantom{\top }}{\mathbf{C}}_{*}^{\phantom{\top }}& ={\mathbf{A}}_{*}^{\phantom{-1}}{\mathbf{B}}_{*}^{-1}{\mathbf{A}}_{*}^{\phantom{-1}}.\hfill \end{array}$$

(16)

If ${\mathbf{A}}_{*}$ is symmetric positive definite, both its square root ${\mathbf{A}}_{*}^{1/2}$ and inverse square root ${\mathbf{A}}_{*}^{-1/2}$ exist. Multiplying both sides of Equation (16) on the left and right by ${\mathbf{A}}_{*}^{-1/2}$ gives

$$\begin{array}{cc}\hfill {\mathbf{A}}_{*}^{-1/2}{\mathbf{C}}_{*}^{\top \phantom{-1/2}}{\mathbf{A}}_{*}^{\phantom{-1/2}}{\mathbf{C}}_{*}^{\phantom{-1/2}}{\mathbf{A}}_{*}^{-1/2}& ={\mathbf{A}}_{*}^{-1/2}{\mathbf{A}}_{*}^{\phantom{-1/2}}{\mathbf{B}}_{*}^{-1\phantom{-1/2}}{\mathbf{A}}_{*}^{\phantom{-1/2}}{\mathbf{A}}_{*}^{-1/2}.\hfill \end{array}$$

Letting $\mathbf{Q}={\mathbf{A}}_{*}^{1/2}{\mathbf{C}}_{*}^{\phantom{-1/2}}{\mathbf{A}}_{*}^{-1/2}$ and hence ${\mathbf{C}}_{*}^{\phantom{-1/2}}={\mathbf{A}}_{*}^{-1/2}\mathbf{Q}{\mathbf{A}}_{*}^{1/2}$, the above expression simplifies to

$$\begin{array}{cc}\hfill {\mathbf{Q}}^{\top }\mathbf{Q}& ={\mathbf{A}}_{*}^{1/2}{\mathbf{B}}_{*}^{-1\phantom{1/2}}{\mathbf{A}}_{*}^{1/2}.\hfill \end{array}$$

The matrix $\mathbf{Q}$ is not unique, since any matrix of the form $\mathbf{R}\mathbf{Q}$, where $\mathbf{R}$ is an orthogonal matrix (${\mathbf{R}}^{\top }\mathbf{R}={\mathbf{I}}_d$), also satisfies the same condition. This phenomenon is known as rotational freedom, and it implies that any matrix square root ${\mathbf{Q}}_{\mathrm{r}}$ is only defined up to an orthogonal rotation or reflection

$$\begin{array}{cc}\hfill {\mathbf{Q}}_{\mathrm{r}}& =\mathbf{R}\mathbf{Q}=\mathbf{R}{\left({\mathbf{A}}_{*}^{1/2}{\mathbf{B}}_{*}^{-1\phantom{1/2}}{\mathbf{A}}_{*}^{1/2}\right)}^{1/2}.\hfill \end{array}$$

Substituting this expression for ${\mathbf{Q}}_{\mathrm{r}}$ back into the expression for ${\mathbf{C}}_{*}$ yields the general form of the $d\times d$ curvature-adjustment matrix from Equation (14)

$$\begin{array}{cc}\hfill {\mathbf{C}}_{*}& ={\mathbf{A}}_{*}^{-1/2}\mathbf{R}{\left({\mathbf{A}}_{*}^{1/2}{\mathbf{B}}_{*}^{-1\phantom{1/2}}{\mathbf{A}}_{*}^{1/2}\right)}^{1/2}{\mathbf{A}}_{*}^{1/2}.\hfill \end{array}$$

(17)

This derivation demonstrates the nonuniqueness of matrix ${\mathbf{C}}_{*}$. If ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$ commute (e.g., are simultaneously diagonalizable), the expression for ${\mathbf{C}}_{*}$ in Equation (17) simplifies to the following more compact form [24,32,38]

$$\begin{array}{cc}\hfill {\mathbf{C}}_{*}^{\phantom{-1}}& ={\mathbf{A}}_{*}^{-1/2}{\mathcal{G}}_0^{1/2}={\mathbf{A}}_{*}^{-1/2}{\mathbf{A}}_{*}^{1/2}{\mathbf{B}}_{*}^{-1/2}{\mathbf{A}}_{*}^{1/2}={\mathbf{B}}_{*}^{-1/2}{\mathbf{A}}_{*}^{1/2},\hfill \end{array}$$

(18)

and enforces the sandwich covariance matrix ${\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{-1}{\mathbf{B}}_{*}^{\phantom{-1}}{\mathbf{A}}_{*}^{-1}$ onto the sampled Markov chains.

The mapping from $\mathsf{\theta }$ to ${\mathsf{\theta }}^{\mathrm{ca}}$ can be regarded as a succession of transformations in which the ellipsoidal contours of ${\mathcal{L}}_n^{\mathrm{ca}}\left(\mathsf{\theta }\right)$ are first mapped to spheroids, and then transformed back to the contours of ${\mathcal{L}}_n\left(\mathsf{\theta }\right)$[38]. The MH acceptance probability of candidate point ${\mathsf{\theta }}_{\mathrm{p}}$ becomes

$$\begin{array}{cc}\hfill P_{\mathrm{acc}}({\mathsf{\theta }}_{(t-1)}\to {\mathsf{\theta }}_{\mathrm{p}})& =min\left\{1,{\displaystyle \frac{P\left({\mathsf{\theta }}_{\mathrm{p}}^{\phantom{\circ }}\right)L_n\left({\mathsf{\theta }}_{\mathrm{p}}^{\mathrm{ca}}\right)q({\mathsf{\theta }}_{(t-1)}\mid {\mathsf{\theta }}_{\mathrm{p}}^{\phantom{\circ }})}{P\left({\mathsf{\theta }}_{(t-1)}^{\phantom{\circ }}\right)L_n\left({\mathsf{\theta }}_{(t-1)}^{\mathrm{ca}}\right)q({\mathsf{\theta }}_{\mathrm{p}}\mid {\mathsf{\theta }}_{(t-1)}^{\phantom{\circ }})}}\right\}.\hfill \end{array}$$

(19)

Thus, the comparison of candidate points ${\mathsf{\theta }}_{\mathrm{p}}$ and the current chain state ${\mathsf{\theta }}_{(t-1)}$ in curvature-adjusted MCMC simulation takes place after ${\mathsf{\theta }}_{\mathrm{p}}$ and ${\mathsf{\theta }}_{(t-1)}$ are scaled and rotated [25]. Ribatet et al. [24] build on a result from Kent [51] to show that the acceptance probability in (19) shares the same asymptotic distribution as the true likelihood ratio. They further argue that the resulting sample has an asymptotic stationary distribution that is normal, with the desired sandwich covariance matrix.

Algorithm 2 outlines the steps of curvature-adjusted MCMC simulation using the MH algorithm. We refer to this procedure as the Curvature-Adjusted Metropolis–Hastings (CAMH) algorithm. The Markov chain generated by the CAMH algorithm has an asymptotic stationary distribution that is d-variate normal with mean ${\widehat{\mathsf{\theta }}}_{*}$ and $d\times d$ sandwich covariance matrix given by ${\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{-1}{\mathbf{B}}_{*}^{\phantom{-1}}{\mathbf{A}}_{*}^{-1}$. For a symmetric proposal distribution, $q({\mathsf{\theta }}_{(t-1)}\mid {\mathsf{\theta }}_{\mathrm{p}})=q({\mathsf{\theta }}_{\mathrm{p}}\mid {\mathsf{\theta }}_{(t-1)})$, and Equation (19) simplifies to

$$P_{\mathrm{acc}}({\mathsf{\theta }}_{(t-1)}\to {\mathsf{\theta }}_{\mathrm{p}})=min\left\{1,{\displaystyle \frac{P\left({\mathsf{\theta }}_{\mathrm{p}}^{\phantom{\circ }}\right)L_n\left({\mathsf{\theta }}_{\mathrm{p}}^{\mathrm{ca}}\right)}{P\left({\mathsf{\theta }}_{(t-1)}^{\phantom{\circ }}\right)L_n\left({\mathsf{\theta }}_{(t-1)}^{\mathrm{ca}}\right)}}\right\}.$$

The above expression further reduces to a likelihood ratio with a uniform prior, $P\left(\mathsf{\theta }\right)={\mathcal{U}}_d({\mathsf{\theta }}^{\min },{\mathsf{\theta }}^{\max })$, where ${\mathsf{\theta }}^{\min }$ and ${\mathsf{\theta }}^{\max }$ are d-vectors with lower and upper bounds of the parameters, where ${\theta }_j^{\min }<{\theta }_j^{\max }$ for all $j=1,\dots ,d$.

The CAMH algorithm facilitates exploration of the posterior sandwich distribution, though it presents certain implementation challenges. First, and foremost, the tuning matrix ${\mathbf{C}}_{*}$ may not be uniquely defined when the log-likelihood ${\mathcal{L}}_n\left(\mathsf{\theta }\right)$ in ${\widehat{\mathsf{\theta }}}_{*}$ is not exactly quadratic. In such cases, the nonuniqueness of the matrix square roots ${\mathbf{A}}_{*}^{1/2}$ and ${\mathbf{B}}_{*}^{1/2}$ introduces an arbitrary rotation of the spheroids prior to the back-transformation. Although this rotation is inconsequential when the likelihood is locally quadratic around ${\widehat{\mathsf{\theta }}}_{*}$, care must be taken to ensure that the mapping preserves directions of asymmetry in the posterior sandwich distribution. This issue is not specific to CAMH; any method that relies on the square roots of the “information” matrices ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$ is subject to this ambiguity.

Second, a subtler issue arises from the nature of the curvature adjustment itself. The transformation is affine and acts on the parameter values such that the likelihood at a point $\mathsf{\theta }$ is evaluated at its transformed counterpart ${\mathsf{\theta }}^{\mathrm{ca}}$, that is, $L_n\left({\mathsf{\theta }}^{\mathrm{ca}}\right)$. This approach may be easy to implement, but is not intuitive. It may assign high likelihoods to points that are relatively far from the MAP estimate ${\widehat{\mathsf{\theta }}}_{*}$, even when their original likelihoods were comparatively low, and vice versa. For example, if ${\mathsf{\theta }}_{\mathrm{m}}$ is a local maximum of $L_n\left(\mathsf{\theta }\right)$, the transformation substitutes $L_n\left({\mathsf{\theta }}_{\mathrm{m}}\right)$ with $L_n\left({\widehat{\mathsf{\theta }}}_{*}+{\mathbf{C}}_{*}({\mathsf{\theta }}_{\mathrm{m}}-{\widehat{\mathsf{\theta }}}_{*})\right)$, regardless of the actual likelihood at ${\mathsf{\theta }}_{\mathrm{m}}$. As a result, asymmetries or non-elliptical features of $L_n\left(\mathsf{\theta }\right)$ can be distorted, compromising the geometric fidelity of the sandwich distribution.

Finally, the parameter transformation used in curvature adjustment can conflict with bounded parameter spaces. A candidate point ${\mathsf{\theta }}_{\mathrm{p}}$ may satisfy the prior constraints in the original parameterization, yet its curvature-adjusted counterpart ${\mathsf{\theta }}_{\mathrm{p}}^{\mathrm{ca}}$ may lie outside the feasible parameter space. This complication is not insurmountable, but requires careful handling to ensure that the Markov chain respects parameter constraints, preserves detailed balance, and maintains acceptable sampling efficiency.

Algorithm 2 Curvature-adjusted Metropolis–Hastings (CAMH)

Input: Prior, $P\left(\mathsf{\theta }\right)$, likelihood, $L_n\left(\mathsf{\theta }\right)$, and transition density, $q({\mathsf{\theta }}_{\mathrm{p}}\mid {\mathsf{\theta }}_{(t-1)})$    Total number of samples T    MAP solution, ${\widehat{\mathsf{\theta }}}_{*}$, and $d\times d$ tuning matrix ${\mathbf{C}}_{*}$ of Equation (18) Output: Samples $\{{\mathsf{\theta }}_{\left(0\right)},{\mathsf{\theta }}_{\left(1\right)},\dots ,{\mathsf{\theta }}_{\left(T\right)}\}$ from sandwich posterior of $P_n\left(\mathsf{\theta }\right)\propto P\left(\mathsf{\theta }\right)L_n\left(\mathsf{\theta }\right)$ Draw initial chain state ${\mathsf{\theta }}_{\left(0\right)}$ from the prior distribution, ${\mathsf{\theta }}_{\left(0\right)}\sim P\left(\mathsf{\theta }\right)$ Transform initial state, ${\mathsf{\theta }}_{\left(0\right)}^{\mathrm{ca}}={\widehat{\mathsf{\theta }}}_{*}+{\mathbf{C}}_{*}({\mathsf{\theta }}_{\left(0\right)}-{\widehat{\mathsf{\theta }}}_{*})$, and compute $L_n\left({\mathsf{\theta }}_{\left(0\right)}^{\mathrm{ca}}\right)$ for  $t=1$  to  T  do     Sample a proposal ${\mathsf{\theta }}_{\mathrm{p}}\sim q(·\mid {\mathsf{\theta }}_{(t-1)})$ from the transition kernel     Transform the candidate point ${\mathsf{\theta }}_{\mathrm{p}}^{\mathrm{ca}}={\widehat{\mathsf{\theta }}}_{*}+{\mathbf{C}}_{*}({\mathsf{\theta }}_{\mathrm{p}}-{\widehat{\mathsf{\theta }}}_{*})$ and compute $L_n\left({\mathsf{\theta }}_{\mathrm{p}}^{\mathrm{ca}}\right)$     Compute the acceptance probability $P_{\mathrm{acc}}({\mathsf{\theta }}_{(t-1)}\to {\mathsf{\theta }}_{\mathrm{p}})$ using Equation (19)     Draw a label Z from a uniform distribution, $Z\sim \mathcal{U}(0,1)$     if $P_{\mathrm{acc}}({\mathsf{\theta }}_{(t-1)}\to {\mathsf{\theta }}_{\mathrm{p}})\ge Z$ then    Accept the candidate point, ${\mathsf{\theta }}_{\left(t\right)}={\mathsf{\theta }}_{\mathrm{p}}$ and $P_n\left({\mathsf{\theta }}_{\left(t\right)}\right)=P_n\left({\mathsf{\theta }}_{\mathrm{p}}\right)$     else    Reject the proposal and set ${\mathsf{\theta }}_{\left(t\right)}={\mathsf{\theta }}_{(t-1)}$ and $P_n\left({\mathsf{\theta }}_{\left(t\right)}\right)=P_n\left({\mathsf{\theta }}_{(t-1)}\right)$     end if end for Return:  $\{{\mathsf{\theta }}_{\left(0\right)},{\mathsf{\theta }}_{\left(1\right)},\dots ,{\mathsf{\theta }}_{\left(T\right)}\}$

4.4. Algorithm 3: Kernel Adjustment

Given the limitations of existing sampling methods, we propose a new approach, so-called kernel-amendment, which combines elements of magnitude- and curvature-adjusted MCMC simulation but introduces two key innovations, (i) a direction-dependent scaling factor that captures asymmetric and non-quadratic features of the sandwich distribution, and (ii) an implementation that avoids matrix square roots of ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$. This method guarantees an accurate description of the sandwich distribution by MCMC methods. We first develop the theoretical framework, then assess the performance of both kernel-amendment and existing MCMC sampling methods through applications to commonly used parametric distributions and to numerical models, using both synthetic and measured data.

Theory

Suppose $P_n\left(\mathsf{\theta }\right)\propto P\left(\mathsf{\theta }\right)L_n\left(\mathsf{\theta }\right)$ is the unnormalized posterior density, or in logarithmic form, ${\mathcal{P}}_n\left(\mathsf{\theta }\right)\propto \mathcal{P}\left(\mathsf{\theta }\right)+{\mathcal{L}}_n\left(\mathsf{\theta }\right)$. Our proposed solution is to sample from the density

$$\begin{array}{cc}\hfill \varphi \left(\mathsf{\theta }\right)& \propto exp\left\{\lambda \left(\mathsf{\theta }\right)\left({\mathcal{L}}_n\left(\mathsf{\theta }\right)-{\mathcal{L}}_n\left({\widehat{\mathsf{\theta }}}_{*}\right)\right)\right\},\hfill \end{array}$$

(20)

where $\lambda \left(\mathsf{\theta }\right):{\mathbb{R}}^d\to \mathbb{R}$ is a scalar-valued function which scales linearly the difference in the log-likelihoods of points $\mathsf{\theta }$ and ${\widehat{\mathsf{\theta }}}_{*}$. The argument of the exponential

$$\begin{array}{cc}\hfill {\mathcal{L}}_n^{\mathrm{p}}(\mathsf{\theta }\mid \lambda )& =log\left(L_n^{\mathrm{p}}(\mathsf{\theta }\mid \lambda )\right)=\lambda \left(\mathsf{\theta }\right)[{\mathcal{L}}_n\left(\mathsf{\theta }\right)-{\mathcal{L}}_n\left({\widehat{\mathsf{\theta }}}_{*}\right)],\hfill \end{array}$$

(21)

is itself a power log-likelihood function. By construction, ${\mathcal{L}}_n^{\mathrm{p}}({\widehat{\mathsf{\theta }}}_{*}\mid \lambda )=0$, while for all other $\mathsf{\theta }\in \mathsf{\Theta }\subseteq {\mathbb{R}}^d$, it is negative. Subtracting the value at ${\widehat{\mathsf{\theta }}}_{*}$ recenters ${\mathcal{L}}_n^{\mathrm{p}}(\mathsf{\theta }\mid \lambda )$ so that its Hessian ${\mathbf{H}}_n\left({\widehat{\mathsf{\theta }}}_{*}\right)={\nabla }^2{\mathcal{L}}_n^{\mathrm{p}}({\widehat{\mathsf{\theta }}}_{*}\mid \lambda )$ accurately captures the local curvature free from arbitrary offsets in ${\mathcal{L}}_n\left(\mathsf{\theta }\right)$. This centering allows us to exactly align the curvature of the power log-likelihood function with that of the sandwich distribution, even when ${\mathcal{L}}_n\left(\mathsf{\theta }\right)$ is asymmetric near the mode and eliminates the need for matrix square roots (as shown later). Choosing ${\widehat{\mathsf{\theta }}}_{*}$ as the centering point ensures generality: under a uniform prior it coincides with the ML estimator, while under an informative prior it becomes the MAP estimator. This makes the approach seamlessly applicable to both frequentist and Bayesian settings. In short, subtracting, ${\mathcal{L}}_n\left({\widehat{\mathsf{\theta }}}_{*}\right)$ normalizes curvature for scaling and avoids contamination by arbitrary log-likelihood offsets.

The Metropolis acceptance probability for a candidate point ${\mathsf{\theta }}_{\mathrm{p}}$ now becomes

$$\begin{array}{cc}\hfill P_{\mathrm{acc}}({\mathsf{\theta }}_{\mathrm{p}}\to {\mathsf{\theta }}_{(t-1)})& =min\left\{1,{\displaystyle \frac{P\left({\mathsf{\theta }}_{\mathrm{p}}\right)L_n^{\mathrm{p}}({\mathsf{\theta }}_{\mathrm{p}}\mid \lambda )q({\mathsf{\theta }}_{(t-1)}\mid {\mathsf{\theta }}_{\mathrm{p}})}{P\left({\mathsf{\theta }}_{(t-1)}\right)L_n^{\mathrm{p}}({\mathsf{\theta }}_{(t-1)}\mid \lambda )q({\mathsf{\theta }}_{\mathrm{p}}\mid {\mathsf{\theta }}_{(t-1)})}}\right\},\hfill \end{array}$$

(22)

where $L_n^{\mathrm{p}}(\mathsf{\theta }\mid \lambda )$ is the normalized power likelihood function

$$\begin{array}{cc}\hfill L_n^{\mathrm{p}}(\mathsf{\theta }\mid \lambda )=exp\left({\mathcal{L}}_n^{\mathrm{p}}(\mathsf{\theta }\mid \lambda )\right)& =exp\left(\lambda \left(\mathsf{\theta }\right)[{\mathcal{L}}_n\left(\mathsf{\theta }\right)-{\mathcal{L}}_n\left({\widehat{\mathsf{\theta }}}_{*}\right)]\right)\hfill \\ \hfill & =exp\left(\lambda \left(\mathsf{\theta }\right)log\left(L_n\left(\mathsf{\theta }\right)\right)-\lambda \left(\mathsf{\theta }\right)log\left(L_n\left({\widehat{\mathsf{\theta }}}_{*}\right)\right)\right)\hfill \\ \hfill & =exp\left(log\left(L_n{\left(\mathsf{\theta }\right)}^{\lambda \left(\mathsf{\theta }\right)}\right)-log\left(L_n{\left({\widehat{\mathsf{\theta }}}_{*}\right)}^{\lambda \left(\mathsf{\theta }\right)}\right)\right)\hfill \\ \hfill & ={\left({\displaystyle \frac{L_n\left(\mathsf{\theta }\right)}{L_n\left({\widehat{\mathsf{\theta }}}_{*}\right)}}\right)}^{\lambda \left(\mathsf{\theta }\right)}.\hfill \end{array}$$

(23)

The acceptance probability in Equation (22) reduces to a likelihood ratio

$$\begin{array}{cc}\hfill P_{\mathrm{acc}}({\mathsf{\theta }}_{\mathrm{p}}\to {\mathsf{\theta }}_{(t-1)})& =min\{1,L_n^{\mathrm{p}}({\mathsf{\theta }}_{\mathrm{p}}\mid \lambda )/L_n^{\mathrm{p}}({\mathsf{\theta }}_{(t-1)}\mid \lambda )\}\hfill \\ \hfill & =min\left\{1,{\displaystyle \frac{L_n{\left({\widehat{\mathsf{\theta }}}_{*}\right)}^{\lambda \left({\mathsf{\theta }}_{(t-1)}\right)}}{L_n{\left({\widehat{\mathsf{\theta }}}_{*}\right)}^{\lambda \left({\mathsf{\theta }}_{\mathrm{p}}\right)}}}{\displaystyle \frac{L_n{\left({\mathsf{\theta }}_{\mathrm{p}}\right)}^{\lambda \left({\mathsf{\theta }}_{\mathrm{p}}\right)}}{L_n{\left({\mathsf{\theta }}_{(t-1)}\right)}^{\lambda \left({\mathsf{\theta }}_{(t-1)}\right)}}}\right\}\hfill \\ \hfill & =min\left\{1,{\displaystyle \frac{L_n{\left({\mathsf{\theta }}_{\mathrm{p}}\right)}^{\lambda \left({\mathsf{\theta }}_{\mathrm{p}}\right)}}{L_n{\left({\mathsf{\theta }}_{(t-1)}\right)}^{\lambda \left({\mathsf{\theta }}_{(t-1)}\right)}}}{L}_n{\left({\widehat{\mathsf{\theta }}}_{*}\right)}^{\left(\lambda \left({\mathsf{\theta }}_{(t-1)}\right)-\lambda \left({\mathsf{\theta }}_{\mathrm{p}}\right)\right)}\right\},\hfill \end{array}$$

in the case of a uniform prior and symmetric transition kernel of the Markov chain.

We are now left to discuss the choice of $\lambda \left(\mathsf{\theta }\right)$. What should this scalar-valued function be? For $\mathsf{\theta }$ in the vicinity of ${\widehat{\mathsf{\theta }}}_{*}$ we know that

$$\begin{array}{cc}\hfill {\mathcal{L}}_n\left(\mathsf{\theta }\right)-{\mathcal{L}}_n\left({\widehat{\mathsf{\theta }}}_{*}\right)& \approx -{\displaystyle \frac{1}{2}}n{(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*})}^{\top }{\mathbf{A}}_{*}(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*}),\hfill \end{array}$$

whereas we desire this to be

$$\begin{array}{cc}\hfill {\mathcal{L}}_n\left(\mathsf{\theta }\right)-{\mathcal{L}}_n\left({\widehat{\mathsf{\theta }}}_{*}\right)& =-{\displaystyle \frac{1}{2}}n{(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*})}^{\top }{\mathbf{A}}_{*}^{\phantom{-1}}{\mathbf{B}}_{*}^{-1}{\mathbf{A}}_{*}^{\phantom{-1}}(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*}).\hfill \end{array}$$

So a sensible choice for $\lambda \left(\mathsf{\theta }\right)$ might be

$$\begin{array}{cc}\hfill & \lambda \left(\mathsf{\theta }\right)={\displaystyle \frac{{(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*})}^{\top }{\mathbf{A}}_{*}^{\phantom{-1}}{\mathbf{B}}_{*}^{-1}{\mathbf{A}}_{*}^{\phantom{-1}}(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*})}{{(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*})}^{\top }{\mathbf{A}}_{*}(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*})}}.\hfill \end{array}$$

(24)

With this formulation for $\lambda \left(\mathsf{\theta }\right)$, the acceptance probability $P_{\mathrm{acc}}({\mathsf{\theta }}_{\mathrm{p}}\to {\mathsf{\theta }}_{(t-1)})$ of Equation (22) will guide a Markov chain to a stationary distribution with the underlying probability density function $\varphi \left(\mathsf{\theta }\right)$ in Equation (20). A formal proof of this result is provided in Appendix B, and the complete recipe for the sandwich-adjusted Metropolis–Hastings (SAMH) algorithm is given in Algorithm 3. By multiplying the difference in the log-likelihoods between any point $\mathsf{\theta }$ and the MAP solution ${\widehat{\mathsf{\theta }}}_{*}$ by the learning rate $\lambda \left(\mathsf{\theta }\right)$, the resulting Markov chain converges to a stationary distribution with the correct asymptotic sandwich variance.

Algorithm 3 Sandwich-adjusted Metropolis–Hastings (SAMH)

Input: Prior, $P\left(\mathsf{\theta }\right)$, likelihood, $L_n\left(\mathsf{\theta }\right)$, and transition density, $q({\mathsf{\theta }}_{\mathrm{p}}\mid {\mathsf{\theta }}_{(t-1)})$    Total number of samples T    MAP solution, ${\widehat{\mathsf{\theta }}}_{*}$, associated likelihood, $L_n\left({\widehat{\mathsf{\theta }}}_{*}\right)$, and matrices ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$ Output: Samples $\{{\mathsf{\theta }}_{\left(0\right)},{\mathsf{\theta }}_{\left(1\right)},\dots ,{\mathsf{\theta }}_{\left(T\right)}\}$ from sandwich posterior of $P_n\left(\mathsf{\theta }\right)\propto P\left(\mathsf{\theta }\right)L_n\left(\mathsf{\theta }\right)$ Draw initial chain state ${\mathsf{\theta }}_{\left(0\right)}$ from the prior distribution, ${\mathsf{\theta }}_{\left(0\right)}\sim P\left(\mathsf{\theta }\right)$ Compute $\lambda \left({\mathsf{\theta }}_{\left(0\right)}\right)$ in Equation (24) and $L_n^{\mathrm{p}}({\mathsf{\theta }}_{\left(0\right)}\mid \lambda )$ of Equation (21) for  $t=1$  to  T  do     Sample a proposal ${\mathsf{\theta }}_{\mathrm{p}}\sim q(·\mid {\mathsf{\theta }}_{(t-1)})$ from the transition density     Compute $\lambda \left({\mathsf{\theta }}_{\mathrm{p}}\right)$ in Equation (24) and $L_n^{\mathrm{p}}({\mathsf{\theta }}_{\mathrm{p}}\mid \lambda )$ in Equation (21)     Compute the acceptance probability, $P_{\mathrm{acc}}({\mathsf{\theta }}_{(t-1)}\to {\mathsf{\theta }}_{\mathrm{p}})$, using Equation (22)     Draw a label Z from a uniform distribution, $Z\sim \mathcal{U}(0,1)$     if $P_{\mathrm{acc}}({\mathsf{\theta }}_{(t-1)}\to {\mathsf{\theta }}_{\mathrm{p}})\ge Z$ then    Accept the candidate point, ${\mathsf{\theta }}_{\left(t\right)}={\mathsf{\theta }}_{\mathrm{p}}$ and $P_n\left({\mathsf{\theta }}_{\left(t\right)}\right)=P_n\left({\mathsf{\theta }}_{\mathrm{p}}\right)$     else    Reject the proposal and set ${\mathsf{\theta }}_{\left(t\right)}={\mathsf{\theta }}_{(t-1)}$ and $P_n\left({\mathsf{\theta }}_{\left(t\right)}\right)=P_n\left({\mathsf{\theta }}_{(t-1)}\right)$     end if end for Return:  $\{{\mathsf{\theta }}_{\left(0\right)},{\mathsf{\theta }}_{\left(1\right)},\dots ,{\mathsf{\theta }}_{\left(T\right)}\}$

Before proceeding with a detailed discussion of our methodology, we briefly revisit our expression for $\lambda \left(\mathsf{\theta }\right)$. According to Definition 7 in Section 2 on Page 5, we can express the numerator of Equation (24) in terms of ${\mathcal{G}}_0$, the expected Godambe information of a single observation

$$\begin{array}{cc}\hfill \lambda \left(\mathsf{\theta }\right)& ={\displaystyle \frac{{(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*})}^{\top }{\mathcal{G}}_0(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*})}{{(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*})}^{\top }{\mathbf{A}}_{*}(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*})}}.\hfill \end{array}$$

(25)

Matrix ${\mathbf{A}}_{*}$ in the denominator is itself an information matrix. Under correct model specification, this matrix equals ${\mathcal{I}}_0$, the expected Fisher information of a single datum. The scalar $\lambda \left(\mathsf{\theta }\right)$ is thus the ratio of two information matrices, with ${\mathbf{A}}_{*}$ reflecting sensitivity-based curvature and ${\mathcal{G}}_0\left(\mathsf{\theta }\right)$ representing the Godambe (sandwich) information. In most practical cases of model misspecification, the latter has smaller curvature, so $0<\lambda \left(\mathsf{\theta }\right)\le 1$, with equality at 1 under correct specification. Values $\lambda \left(\mathsf{\theta }\right)>1$ can occur but are uncommon, arising when the variability-based curvature locally exceeds the sensitivity-based curvature. In all cases, $\lambda \left(\mathsf{\theta }\right)$ acts as a parameter-dependent learning rate that tempers data informativeness, ensuring that Algorithm 3 converges to the sandwich rather than the naive posterior distribution. The nonnegative multiplier $\lambda \left(\mathsf{\theta }\right)$ can also be viewed as a kernel: it is symmetric about ${\widehat{\mathsf{\theta }}}_{*}$ and positive semi-definite due to the information matrices in both numerator and denominator. In keeping with the terminology of the other two methods, we refer to our approach as the kernel-adjustment method, though the term generalized power likelihood is also appropriate.

Remark 5. 

We could define $\varphi \left(\mathsf{\theta }\right)=exp\left(-\frac{1}{2}n{(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*})}^{\top }{\mathbf{A}}_{*}^{\phantom{-1}}{\mathbf{B}}_{*}^{-1}{\mathbf{A}}_{*}^{\phantom{-1}}(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*})\right)$ as the probability density function of the sandwich posterior distribution [52]. This mathematical form is equivalent to a d-variate normal distribution ${\mathcal{N}}_d({\widehat{\mathsf{\theta }}}_{*},{\displaystyle \frac{1}{n}}{\mathbf{A}}_{*}^{\phantom{-1}}{\mathbf{B}}_{*}^{-1}{\mathbf{A}}_{*}^{\phantom{-1}})$ centered on ${\widehat{\mathsf{\theta }}}_{*}$ with $d\times d$ sandwich covariance matrix. We can draw any desired number of “posterior” samples from this distribution. However, this assumes that the local quadratic sandwich approximation is valid. Algorithm 3 relaxes this assumption for finite n. The rationale for choosing a multiplicative form—rather than, say, an additive one—will become evident soon when we interpret our method in terms of power likelihoods.

Remark 6. 

For the single-parameter case, matrices ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$ are scalars, and $\lambda ={\mathbf{A}}_{*}/{\mathbf{B}}_{*}$ can be interpreted as an estimator of the “best” λ in power likelihood, $exp\left(\lambda {\mathcal{L}}_n\left(\mathsf{\theta }\right)\right)$, in the sense of having correctly sized credible sets asymptotically. In other words, for $d=1$, our approach reduces to the magnitude adjustment method (Method 1), but with one important distinction. We apply the power λ to the score difference ${\mathcal{L}}_n\left(\mathsf{\theta }\right)-{\mathcal{L}}_n\left({\widehat{\mathsf{\theta }}}_{*}\right)$ rather than to the log-likelihood ${\mathcal{L}}_n\left(\mathsf{\theta }\right)$ itself, as in Ribatet et al. [24]. This centering of the power likelihood around ${\mathcal{L}}_n\left({\widehat{\mathsf{\theta }}}_{*}\right)$ ensures proper scaling for $d>1$.

Remark 7. 

The ratio $\lambda \left(\mathsf{\theta }\right)$ in Equation (24) depends only on the direction of $\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*}$, not on its magnitude. In fact, we obtain $\lambda \left(\mathsf{\theta }\right)=\lambda \left(r\mathsf{\theta }\right)$ for any scalar $r\ne 0$. Although $\lambda \left(\mathsf{\theta }\right)$ is not defined at $\mathsf{\theta }={\widehat{\mathsf{\theta }}}_{*}$, the product $\lambda \left(\mathsf{\theta }\right)[{\mathcal{L}}_n\left(\mathsf{\theta }\right)-{\mathcal{L}}_n\left({\widehat{\mathsf{\theta }}}_{*}\right)]$ remains well defined. This is because both the denominator of Equation (24) and the log-likelihood difference ${\mathcal{L}}_n\left(\mathsf{\theta }\right)-{\mathcal{L}}_n\left({\widehat{\mathsf{\theta }}}_{*}\right)$ exhibit similar quadratic behavior in a neighborhood of $\mathsf{\theta }={\widehat{\mathsf{\theta }}}_{*}$. Thus, Algorithm 3 generalizes the power likelihood to the multi-parameter case $d>1$ as

$$P_n\left(\mathsf{\theta }\right)\propto exp\left(\lambda \left(\mathsf{\theta }\right)\left({\mathcal{L}}_n\left(\mathsf{\theta }\right)-{\mathcal{L}}_n\left({\widehat{\mathsf{\theta }}}_{*}\right)\right)\right),$$

where $\lambda \left(\mathsf{\theta }\right)=\lambda \left(r\mathsf{\theta }\right)$ for any scalar $r\ne 0$, thus defining a distinct scaling factor for each direction in parameter space.

Remark 8. 

The kernel $\lambda \left(\mathsf{\theta }\right)$ has an eigenspace decomposition. We can transform the $d\times 1$-vector of differences $\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*}$ and write lambda as a function of the matrix-vector product $\mathsf{\vartheta }={\mathbf{A}}_{*}^{1/2}(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*})$. The resulting expression

$${\lambda }_{\mathsf{\vartheta }}\left(\mathsf{\vartheta }\right)={\displaystyle \frac{{\mathsf{\vartheta }}^{\top }{\mathbf{A}}_{*}^{1/2}{\mathbf{B}}_{*}^{-1}{\mathbf{A}}_{*}^{1/2}\mathsf{\vartheta }}{{\mathsf{\vartheta }}^{\top }\mathsf{\vartheta }}},$$

is a Rayleigh quotient in terms of $\mathsf{\vartheta }$. The largest and smallest value of the Rayleigh quotient are equal to the largest ${\underline{\lambda }}_1$ and smallest ${\underline{\lambda }}_d$ eigenvalues of the precision matrix $\mathbf{M}\equiv {\mathbf{A}}_{*}^{1/2\phantom{-1}}{\mathbf{B}}_{*}^{-1\phantom{1/2}}{\mathbf{A}}_{*}^{1/2\phantom{-1}}$.

Remark 9. 

Since $\mathbf{M}$ is the inverse variance-covariance matrix of the posterior of $\mathsf{\vartheta }$, the critical values of ${\lambda }_{\mathsf{\vartheta }}\left(\mathsf{\vartheta }\right)$ where the gradient vanishes correspond to the orthogonal eigenvectors ${\vartheta }_1\le \dots \le {\vartheta }_d$ of $\mathbf{M}$.

Remark 10. 

Eigenvalues and eigenvectors of $\mathbf{M}$ are informative on how much information the data carry on average on the transformed pseudo-true parameter values ${\mathsf{\vartheta }}_{*}={\mathbf{A}}_{*}^{1/2}({\mathsf{\theta }}_{*}-\widehat{\mathsf{\theta }})$ in each of the d eigendirections of $\mathsf{\vartheta }$ relative to what you would expect under correct specification.

4.5. Other Methods

As a general remedy for poor uncertainty quantification under misspecification, Frazier et al. [53] replace the usual posterior with a score-based approximate posterior

$${\tilde{P}}_n\left(\mathsf{\theta }\right)\propto P\left(\mathsf{\theta }\right)exp\left\{-{\displaystyle \frac{n}{2}}{\overline{\mathbf{s}}}_n^{\top }\left(\mathsf{\theta }\right){\widehat{\mathbf{B}}}_n^{-1}\left(\mathsf{\theta }\right){\overline{\mathbf{s}}}_n^{\phantom{\top }}\left(\mathsf{\theta }\right)\right\},$$

where ${\overline{\mathbf{s}}}_n\left(\mathsf{\theta }\right)={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n{\mathbf{s}}_{{\omega }_i}\left(\mathsf{\theta }\right)$ is the $d\times 1$ mean score with ${\mathbf{s}}_{{\omega }_i}\left(\mathsf{\theta }\right)=\nabla {\mathcal{L}}_{{\omega }_i}\left(\mathsf{\theta }\right)$, and ${\widehat{\mathbf{B}}}_n\left(\mathsf{\theta }\right)$ estimates the score variability. This works well as $n\to \infty$, but with a flat prior the exponential kernel equals 1 at any $\mathsf{\theta }$ where ${\overline{\mathbf{s}}}_n\left(\mathsf{\theta }\right)={\mathbf{0}}_d$, so all such roots attain the same peak height. This obscures the relative support of competing modes and distorts the faithful representation of multimodal posteriors. These issues are most acute at small sample sizes.

In a recent paper, Li and Rice [54] reviewed Bayesian analogues of sandwich variance estimators and derive Bayes rules under a so-called balanced inference loss function, ${\mathsf{\ell }}_{\mathrm{BI}}\left(\mathsf{\theta }\right)$. Such loss functions, originally introduced by Zellner [55] and discussed by Dawid and Sebastiani [56] in the context of Bayesian decision theory and optimal experimental design, blend attributes of standard parametric inference with weighted average penalty terms for lack-of-fit and estimation error

$$\begin{array}{cc}\hfill {\mathsf{\ell }}_{\mathrm{BI}}(\mathsf{\theta },{\mathsf{\Sigma }}_n,\mathsf{\Phi })& =log\left(\right|{\mathsf{\Sigma }}_n\left|\right)+\underset{\mathrm{Estimation}\mathrm{error}}{\underbrace{{(\mathsf{\theta }-{\mathsf{\mu }}_{\mathsf{\theta }})}^{\top }\mathsf{\Phi }{\mathsf{\Sigma }}_n^{-1}(\mathsf{\theta }-{\mathsf{\mu }}_{\mathsf{\theta }})}}+\underset{\mathrm{Lack}\mathrm{of}\mathrm{fit}}{\underbrace{{\displaystyle \frac{1}{n}}\sum _{i=1}^n{\nabla }^{\top }{\mathcal{L}}_{{\omega }_i}\left(\mathsf{\theta }\right){\left\{\mathsf{\Phi }{\mathbf{A}}_n\left(\mathsf{\theta }\right)\right\}}^{-1}\nabla {\mathcal{L}}_{{\omega }_i}\left(\mathsf{\theta }\right)}},\hfill \end{array}$$

(26)

where ${\mathsf{\Sigma }}_n$ signifies the $n\times n$ measurement error covariance matrix of the data, ${\omega }_1,\dots ,{\omega }_n$, ${\mathsf{\mu }}_{\mathsf{\theta }}$ is the $d\times 1$ vector of expected parameter values for the n data points, and $\mathsf{\Phi }$ is a $d\times d$ positive definite weighting matrix. This balanced inference loss function is equivalent to a negative log-likelihood function in a Bayesian context. Li and Rice [54] show by simulation that the balanced inference loss function yields robust Bayesian standard error estimates under model misspecification, thus retaining the attractive features of frequentist inference. Yet, the balanced loss function of Equation (26) is optimal only when residuals follow a Gaussian distribution. This is a significant limitation for discharge residuals of conceptual hydrologic models, which typically deviate from normality and are more accurately described by Laplacian or double-exponential distributions [11,57]. Moreover, the balanced loss function requires repeated evaluation of the sensitivity matrix ${\mathbf{A}}_n$—i.e., the empirical Fisher information, which incurs a substantial computational overhead on the order of $d^2$ model evaluations for each MH candidate point ${\mathsf{\theta }}_{\mathrm{p}}$. The balanced inference loss function is well-suited for the ABC model used in our first sandwich paper [8], as it provides an analytic expression for the sensitivity matrix ${\mathbf{A}}_n$. For other studies, we resort instead to magnitude-, curvature-, or sandwich-adjusted MCMC simulation.

5. Empirical Estimates of Information Matrices

Sandwich-adjusted MCMC simulation assumes knowledge of the true parameter values ${\mathsf{\theta }}_0$ (and ${\mathsf{\theta }}_{*}$ under misspecification) and the sensitivity and variability matrices ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$, respectively, of the data-generating process $\mathfrak{S}$. These are theoretical quantities that are not known in practice. In Section 2, we define that ${\mathbf{A}}_{*}$ is the probability limit of ${\mathbf{A}}_n$ and, similarly, ${\mathbf{B}}_{*}=plim{\mathbf{B}}_n$, where ${\mathbf{A}}_n$ and ${\mathbf{B}}_n$ are the averages of the sensitivity and variability matrices for the n data points ${\omega }_1,\dots ,{\omega }_n$. Thus, we must replace the population quantities ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$ with their sample-based estimates

$${\mathbf{A}}_n=-{\mathbb{P}}_{{\widehat{\mathsf{\theta }}}_{*}}^n{\nabla }^2{\mathcal{L}}_{\omega }\left(\mathsf{\theta }\right)\mathrm{and}{\mathbf{B}}_n={\mathbb{P}}_{{\widehat{\mathsf{\theta }}}_{*}}^n\nabla {\mathcal{L}}_{\omega }\left(\mathsf{\theta }\right){\nabla }_{\mathsf{\theta }}^{\top }{\mathcal{L}}_{\omega }\left(\mathsf{\theta }\right),$$

where the notation with a precursor ${\mathbb{P}}_{{\widehat{\mathsf{\theta }}}_{*}}^n$ is borrowed from Kleijn and van der Vaart [9] and designates that we must evaluate the n-sample average for ${\omega }_1,\dots ,{\omega }_n$ of the quantity to its right and in the MAP solution $\mathsf{\theta }={\widehat{\mathsf{\theta }}}_{*}$. Sample-based quantities exhibit variability due to the random nature of the data, but should be consistent estimates. That means that as the sample size n increases, ${\mathbf{A}}_n$ and ${\mathbf{B}}_n$ tend to converge in probability to ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$, respectively. The matrices ${\mathbf{C}}_{*}$ and ${\mathsf{\Psi }}_{*}$ used in a priori and a posteriori curvature-adjusted posterior exploration, respectively, are replaced by their sample equivalents

$${\mathbf{C}}_n={\mathbf{B}}_n^{-1/2}{\mathbf{A}}_n^{1/2}\mathrm{and}{\mathsf{\Psi }}_n={\mathbf{A}}_n^{-1}{\mathbf{B}}_n^{1/2}{\mathbf{A}}_n^{1/2},$$

where the matrix square roots ${\mathbf{A}}_n^{1/2}$ and ${\mathbf{B}}_n^{1/2}$ follow from Equation (13) using singular value decomposition.

The MAP solution can be determined from an optimization method or a MCMC pre-trial. The Markov chain sample that maximizes the posterior density, $P_n\left(\mathsf{\theta }\right)=P\left(\mathsf{\theta }\right)L_n\left(\mathsf{\theta }\right)$, is a MAP estimator. Next, the $d\times 1$ vector of first-order derivatives, $\nabla {\mathcal{L}}_{\omega }\left({\mathsf{\theta }}_{*}\right)$, and $d\times d$ matrix of second-order derivatives, ${\nabla }^2{\mathcal{L}}_{\omega }\left({\mathsf{\theta }}_{*}\right)$, can be determined by numerical means using values of the log-likelihood function ${\mathcal{L}}_n\left(\mathsf{\theta }\right)$ at points nearby ${\widehat{\mathsf{\theta }}}_{*}$. We use the DERIVESTsuite toolbox of D’Errico [58], a Matlab collection of fully adaptive numerical differentiation methods for scalar- and vector-valued functions. This toolbox handles the computation of first- and higher-order derivatives of functions that do not have simple analytical expressions. We employ semi-adaptive central difference schemes of varying orders, combined with a generalized Richardson [59] extrapolation approach (this method is also referred to as multi-term extrapolation in the context of numerical integration Romberg [60]), to enhance the accuracy of the first- and second-order partial derivatives of the log-likelihood function ${\mathcal{L}}_n\left(\mathsf{\theta }\right)$ w.r.t. the parameters. This estimation is obtained using a sequence of logarithmically spaced points away from the MAP solution. The “best” differencing interval is automatically selected from the sequence of proportionally cascading points to minimize the approximation errors of $\nabla {\mathcal{L}}_n\left({\mathsf{\theta }}_{*}\right)$ and ${\nabla }^2{\mathcal{L}}_n\left({\mathsf{\theta }}_{*}\right)$.

If successive data points ${\omega }_1,\dots ,{\omega }_n$ exhibit serial correlation, then we must correct the variability matrix ${\mathbf{B}}_n$ for possible autocorrelation among the successive scores, $\nabla {\mathcal{L}}_{{\omega }_1}\left({\widehat{\mathsf{\theta }}}_{*}\right),\dots ,\nabla {\mathcal{L}}_{{\omega }_n}\left({\widehat{\mathsf{\theta }}}_{*}\right)$. As in Vrugt et al. [8], we use the estimator of Newey and West [61] to determine the variability matrix ${\mathsf{\beta }}_n$ of the scores ${\mathbf{g}}_t=\nabla {\mathcal{L}}_{{\omega }_t}\left({\widehat{\mathsf{\theta }}}_{*}\right)$ as follows

$$\begin{array}{cc}\hfill {\mathsf{\beta }}_n& ={\mathfrak{B}}_0+\sum _{\tau =1}^{{\tau }_{\max }}w(\tau ,{\tau }_{\max })({\mathfrak{B}}_{\tau }+{\mathfrak{B}}_{\tau }^{\top }),\hfill \end{array}$$

(27)

where

$$\begin{array}{cc}\hfill {\mathfrak{B}}_{\tau }& ={\displaystyle \frac{1}{n}}\sum _{t=\tau +1}^n{\mathbf{g}}_t{\mathbf{g}}_{t-\tau }^{\top },\hfill \end{array}$$

is an estimate of the autocovariance matrix of scores a distance $\tau$ apart, ${\tau }_{\max }\in {\mathbb{N}}_{+}$ signifies the maximum lag and

$$\begin{array}{cc}\hfill w(\tau ,{\tau }_{\max })& =1-{\displaystyle \frac{\tau }{1+{\tau }_{\max }}}\tau \in [0,{\tau }_{\max }],\hfill \end{array}$$

is a weight function which smooths the sample autocovariance function [62]. For $\tau =0$, we yield ${\mathfrak{B}}_0={\displaystyle \frac{1}{n}}{\sum }_{t=1}^n{\mathbf{g}}_t{\mathbf{g}}_t^{\top }$, which corresponds to ${\mathbf{B}}_n$, the variance matrix of the scores—provided that the scores have zero mean. Under correct model specification, the sum of the lagged autocovariance matrices, ${\mathfrak{B}}_{\tau }$ vanishes, yielding a $d\times d$ zero matrix. Consequently, we have ${\mathsf{\beta }}_n={\mathbf{B}}_n={\mathfrak{B}}_0$.

Bartlett [63] proposed truncating the sum in Equation (27) at a finite lag ${\tau }_{\max }$ so as to balance the trade-off between estimator variance and bias. This finite lag is also called the Bartlett window. Larger windows increase the estimator’s variance, whereas smaller values of ${\tau }_{\max }$ increase the bias of ${\mathsf{\beta }}_n$ by omitting relevant score autocovariances. Bartlett’s ideas about the adequate choice of the truncation lag ${\tau }_{\max }$ have been formalized in rules of thumb such as ${\tau }_{\max }=\lfloor c·n^{1/4}\rfloor$ or ${\tau }_{\max }=c\lfloor 4{(n/100)}^{2/9}\rfloor$ [64], where c is a small positive integer and the function $\lfloor ·\rfloor$ rounds down to the nearest integer. If the scores exhibit strong autocorrelation, one can set c relatively large, say $c=10$, otherwise, one may use $c=1$. We set $c=5$ and yield values of ${\tau }_{\max }$ on the order of 12, 18, 32, and 55 for data sets of length $n=10$, $n=100$, $n=1000$, and $n=\mathrm{10,000}$, respectively.

6. Case Studies

We demonstrate the different sampling methods by application to three different case studies with increasing complexity. The first two case studies involve statistical models and analytic differentiation. These two studies are purposely kept simple, as this allows us to clearly demonstrate the effects of model misspecification and illustrate how the sandwich estimator rectifies resulting biases in uncertainty quantification as a result of a wrong model parameterization (study 1) or an inadequate parametric form (study 2) for the data-generating process. The third and last study considers the application of the presented methods to rainfall-discharge simulation using the Xinanjiang model [65,66]. This study confirms that traditional MCMC methods produce overly narrow credible regions, so-called overconditioning, and demonstrates the advantages of our proposed SAMH algorithm (Algorithm 3) for sandwich-adjusted posterior exploration.

For MCMC simulation, we employ the DREAM_(ZS) algorithm [67,68,69], a differential evolution-based sampler that evolves multiple chains in parallel. Candidate points are generated dynamically using linear combinations of differences between chain states. The transition kernel is self-adaptive, automatically adjusting to the scale and orientation of the target distribution, $P_n\left(\mathsf{\theta }\right)\propto P\left(\mathsf{\theta }\right)L_n\left(\mathsf{\theta }\right)$. Computational efficiency is not a primary concern in this paper, as our main objective is to evaluate the theoretical and practical differences between the magnitude-, curvature-, and sandwich-adjusted MCMC simulation methods. The relative speed with which an MCMC method converges to the sandwich distribution does not influence the validity of these adjustments, which are the central focus of our work. Nonetheless, the DREAM algorithm has been benchmarked extensively and shown to perform well across a wide range of complex inference problems (see references in Vrugt [28]). To ensure reliable posterior exploration, we monitor convergence using a suite of established diagnostics, including the single-chain methods of Raftery and Lewis [70] and Geweke [71], as well as the multi-chain scale-reduction factors proposed by Gelman and Rubin [72] and Brooks and Gelman [73], following best practices advocated by Cowles and Carlin [74].

6.1. Case Study 1

We revisit example 1 in Section 3 and compute the ML estimate $\widehat{m}$ of the mean of the normal distribution model $N(m,s^2)$, and the corresponding values of ${\mathbf{A}}_n$, ${\mathbf{B}}_n$, omnibus scalar $\widehat{k}={\mathbf{B}}_n^{-1}{\mathbf{A}}_n$, naive ${\mathsf{\Sigma }}_n^{\mathrm{naive}}={\displaystyle \frac{1}{n}}{\mathbf{A}}_n^{-1}$, and sandwich ${\mathsf{\Sigma }}_n^{\mathrm{sand}}={\displaystyle \frac{1}{n}}{\mathbf{A}}_n^{-1}{\mathbf{B}}_n{\mathbf{A}}_n^{-1}$ variances using $s^2=2$, $s^2=1$ and $s^2=1/2$. The ML solution $\widehat{m}$ is simply equal to the sample mean of the data points ${\omega }_1,\dots ,{\omega }_{100}$ drawn from $\mathcal{N}(\mu ,{\sigma }^2)$ with $\mu =0$ and ${\sigma }^2=1$ and ${\mathbf{A}}_n$ and ${\mathbf{B}}_n$ are derived by numerical means using the DERIVESTsuite toolbox of D’Errico [58]. We repeat this computation for $M={10}^4$ different realizations of the n data points.

Table 1. Normal distribution model $N(m,s^2)$: ML estimate of $\widehat{m}$, and associated values of the information matrices ${\mathbf{A}}_n$ and ${\mathbf{B}}_n$, omnibus scalar $\widehat{k}$, and naive and sandwich variance estimators using the normal log-likelihood function ${\mathcal{L}}_n^{\mathrm{n}}(m\mid s^2)$ in Equation (5) with $s^2=2$, $s^2=1$, and $s^2=0.5$. Tabulated values are an average of $M={10}^4$ different realizations of ${\omega }_1,\dots ,{\omega }_{100}$ sampled from the data generating process. Standard deviations are listed between parenthesis.

${\mathit{s}}^2$	$\widehat{\mathit{m}}$	${\mathbf{A}}_{\mathit{n}}$	${\mathbf{B}}_{\mathit{n}}$	$\widehat{\mathit{k}}$	${\mathsf{\Sigma }}_{\mathit{n}}^{\mathbf{naive}}$	${\mathsf{\Sigma }}_{\mathit{n}}^{\mathbf{sand}}$
2	−0.001 (0.099)	0.500 (0.000)	0.247 (0.036)	2.041 (0.298)	0.020 (0.000)	0.010 (0.001)
1	−0.001 (0.099)	1.000 (0.000)	0.990 (0.143)	1.021 (0.149)	0.010 (0.000)	0.010 (0.001)
0.5	−0.001 (0.099)	2.000 (0.000)	3.959 (0.568)	0.516 (0.089)	0.005 (0.000)	0.010 (0.001)

presents the result of this Monte Carlo experiment and lists mean values of $\widehat{m}$, ${\mathbf{A}}_n$, ${\mathbf{B}}_n$, $\widehat{k}$, ${\mathsf{\Sigma }}_n^{\mathrm{naive}}$ and ${\mathsf{\Sigma }}_n^{\mathrm{sand}}$ and their respective standard deviations (in parentheses). The Matlab code is given in Appendix C.

The tabulated results confirm the theory. The ML estimate of the mean $\widehat{m}$ is centered around zero and has a standard deviation that approaches the theoretic standard deviation $\sqrt{{\sigma }^2/n}=0.1$. The ML sensitivity matrix ${\mathbf{A}}_n$ derived from numerical differentiation equals its theoretic value ${\mathbf{A}}_{*}=s^{-2}$ and does not differ between the trials. The variability matrix ${\mathbf{B}}_n$ approaches its theoretic value ${\mathbf{B}}_{*}=s^{-4}{\sigma }^2$ and has a nonzero standard deviation as ${\sigma }^2$ is replaced by the sample variance of the $\omega$’s. The mean of the omnibus scalar approaches its theoretic value $k={\mathbf{B}}_{*}^{-1}{\mathbf{A}}_{*}=s^4{\sigma }^{-2}{s}^{-2}$. Thus, we find that $k=s^2{\sigma }^{-2}$ and the standard deviation between parenthesis is a result that ${\sigma }^2$ of the data-generating process is replaced by the sample variance of the $\omega$’s. The naive variance estimator is equal to its theoretic value ${\mathsf{\Sigma }}_n^{\mathrm{naive}}=s^2/n$ and does not differ between the trials, as n and $s^2$ are fixed. The sandwich variance estimator ${\mathsf{\Sigma }}_n^{\mathrm{sand}}$ does not depend on the value of $s^2$ and asymptotically converges to the true variance of the mean $\mu$ of the data-generating process. The standard deviation is the result of the variation in ${\mathbf{B}}_n$ in the Monte Carlo trials.

Figure 3 displays the histograms of the omnibus scalar k for each of the three normal distribution models. The use of $k{\mathcal{L}}_n\left(\mathsf{\theta }\right)$ will retrieve the sandwich variance ${\mathsf{\Sigma }}_n^{\mathrm{sand}}$.

When $s^2=2$, the model underestimates the information contained in the data and the omnibus scalar is greater than one. Vice versa, for $s^2=0.5$ we systematically overestimate the informativeness of the data and, as a result, $k<1$, to slow down learning and produce robust confidence intervals for $\mu$, the mean of the data-generating process. For $s^2=1$, the normal distribution model $N(m,s^2)$ is equal to the standard normal distribution $\mathcal{N}(0,1)$ of the data generating process and $k=1$.

To better understand the relationship between the number n of data points $\omega$ and the naive and sandwich variances, we repeat the analysis of Table 1 for different values of n. Figure 4 presents the results of this analysis.

The naive variance decreases linearly on a logarithmic scale with the length n of the training record. The slope of this line is proportional to $1/n$ on a linear scale. The naive variance of m depends on the choice of $s^2$. The sandwich variance does not depend on $s^2$ and settles on the true variance of m with increasing number of data points $\omega$.

Table 2. Coverage in % of the true mean $\mu$ of the data generating process $\mathcal{N}(\mu ,{\sigma }^2)$ for $100(1-\alpha )$% confidence intervals of the ML estimate $\widehat{m}$ under the normal distribution model $N(m,s^2)$ with $s^2=2$, $s^2=1$, and $s^2=1/2$ using the naive and sandwich variance estimators. Results are based on the code in Appendix C using $M={10}^3$ trials with $\mu =0$, ${\sigma }^2=1$, and sample size $n=100$.

$\mathit{\alpha }$	${\mathit{s}}^2=2$		${\mathit{s}}^2=1$		${\mathit{s}}^2=0.5$		Theoretic Coverage
	${\mathsf{\Sigma }}_{\mathbf{n}}^{\mathbf{naive}}$	${\mathsf{\Sigma }}_{\mathbf{n}}^{\mathbf{sand}}$	${\mathsf{\Sigma }}_{\mathbf{n}}^{\mathbf{naive}}$	${\mathsf{\Sigma }}_{\mathbf{n}}^{\mathbf{sand}}$	${\mathsf{\Sigma }}_{\mathbf{n}}^{\mathbf{naive}}$	${\mathsf{\Sigma }}_{\mathbf{n}}^{\mathbf{sand}}$
0.01	100.00	98.70	98.90	98.70	93.00	98.70	99%
0.05	99.10	94.80	94.70	94.80	83.10	94.80	95%
0.10	97.90	90.00	90.30	90.00	75.70	90.00	90%
0.20	92.70	79.20	79.20	79.20	64.00	79.20	80%
0.30	84.90	71.60	71.40	71.60	53.40	71.60	70%
0.40	75.90	60.10	60.40	60.00	44.00	60.00	60%
0.50	67.00	49.70	49.80	49.80	34.50	49.80	50%

examines the coverage probabilities of the true mean $\mu$ of the data generating process according to the $100(1-\alpha )$% confidence intervals of $\widehat{\mu }$ derived from the naive and sandwich variance estimators.

The results of Table 2 demonstrate that the sandwich variance estimator provides adequate confidence intervals of the mean $\mu$ of the data-generating process, even if the underlying model is misspecified. The sandwich estimator ${\mathsf{\Sigma }}_n^{\mathrm{sand}}$ consistently achieves the correct frequentist coverage probabilities, whereas the naive variance estimator ${\mathsf{\Sigma }}_n^{\mathrm{naive}}$ either over- ($s^2=2$) or under- ($s^2=1/2$) estimates the coverage probabilities. The confidence intervals are either too dispersed or too sharp.

This concludes our first case study. This study was rather unrealistic in that misspecification was introduced by affixing one of the model parameters $s^2$ to a wrong value. The correct distribution was used, but with a wrong value for one of its parameters, namely the variance $s^2$. In the next study, we are going to take misspecification one step further and use a different model for inference than was used to generate the data.

6.2. Case Study 2

Our second case study is another analytic exercise, but one that better reflects practice as the parametric form of our model differs from that of the data-generating process. We draw n measurements ${\omega }_1,\dots ,{\omega }_n$ from a gamma distribution $\mathsf{\Omega }\sim \mathcal{G}(a,b)$ with pdf

$$f_{\mathcal{G}}(\omega \mid a,b)={\displaystyle \frac{1}{b^a\mathsf{\Gamma }\left(a\right)}}{\omega }^{a-1}exp(-\omega /b),\omega \ge 0,$$

(28)

where $a>0$ and $b>0$ are shape and scale parameters, respectively, and $\mathsf{\Gamma }\left(z\right)$ is the Gamma function. Now, suppose our model for $\mathsf{\Omega }$ is not a gamma but an exponential distribution $\mathcal{E}\left(\mu \right)$ with location parameter $\mu >0$ and pdf

$$f_{\mathcal{E}}(\omega \mid \mu )={\mu }^{-1}exp(-\omega /\mu ),\omega \ge 0.$$

Note that it is not uncommon to parameterize $\mathcal{E}\left(\mu \right)$ with a rate parameter $\lambda ={\mu }^{-1}$ instead. The likelihood function for a single observation $\omega$ is now equal to

$$L_{\omega }\left(\mu \right)=f(\omega \mid \mu )={\mu }^{-1}exp(-\omega /\mu ),$$

and the log-likelihood ${\mathcal{L}}_{\omega }\left(\mu \right)$ becomes

$${\mathcal{L}}_{\omega }\left(\mu \right)=log\left(L_{\omega }\left(\mu \right)\right)=-log\left(\mu \right)-\omega /\mu .$$

Appendix D derives analytic expressions for the naive and sandwich variance estimators of the mean of the exponential distribution. We yield

$${\mathsf{\Sigma }}_n=\left\{\begin{array}{cc}{\mathsf{\Sigma }}_n^{\mathrm{naive}}=m_{\omega }^2/n={\widehat{\mathcal{I}}}_n^{-1},\hfill & \mathrm{naive}\mathrm{variance},\hfill \\ {\mathsf{\Sigma }}_n^{\mathrm{sand}}=s_{\omega }^2/n={\widehat{\mathcal{G}}}_n^{-1},\hfill & \mathrm{sandwich}\mathrm{variance}.\hfill \end{array}\right.$$

(29)

where $m_{\omega }$ and $s_{\omega }^2$ are the sample mean and sample variance of the data points, ${\omega }_1,\dots ,{\omega }_n$. According to Equation (A12), the omnibus scalar k is now equal to $\widehat{k}=m_{\omega }^2{s}_{\omega }^{-2}$, whereas its theoretical value, derived in Equation (A13), corresponds to the shape parameter a of $\mathcal{G}(a,b)$.

Table 3. Coverage in % of the true mean $\mu =a·b$ of the data generating process $\mathcal{G}(a,b)$ for $100(1-\alpha )$% confidence intervals of the ML estimate $\widehat{\mu }$ under the exponential distribution $\mathcal{E}\left(\mu \right)$ using the naive and sandwich variance estimators. Results are based on $M={10}^4$ trials with $a=0.5$, $b=0.2$ and sample size $n=100$. The Matlab code is given in Appendix D.

Method/Coverage	$\mathit{\alpha }=0.01$	$\mathit{\alpha }=0.05$	$\mathit{\alpha }=0.1$	$\mathit{\alpha }=0.2$	$\mathit{\alpha }=0.3$	$\mathit{\alpha }=0.4$	$\mathit{\alpha }=0.5$
	99%	95%	90%	80%	70%	60%	50%
Naive estimator	92.98	83.56	75.62	63.71	54.22	45.26	37.19
Sandwich estimator	97.71	93.71	88.46	79.04	68.73	59.38	49.72

confirms again the erroneous description of the confidence intervals by the naive variance estimator. The $100(1-\alpha )$% confidence intervals are too sharp and underestimate the theoretic coverage probabilities of the mean $\mu$ of the data generating process. This overconditioning is a result of misspecification and, thus, due to a misalignment of the sensitivity and variability matrices. In contrast to other methods, the coverage probabilities of the sandwich estimator align much more closely with theoretical expectations. The estimates are not perfect, as a result of the symmetry assumption used in constructing confidence intervals for $\widehat{\mu }$. This assumption is not valid for the exponential distribution and is further exacerbated when the sample size n is small. To mitigate this latter effect, we chose $n=100$ in our Monte Carlo experiments. To address the asymmetry, one could construct non-symmetric confidence intervals by identifying the shortest interval for $\widehat{\mu }$ that contains the true mean with probability $1-\alpha$. However, doing so would require knowledge of the posterior distribution of $\widehat{\mu }$, which is generally not available in frequentist settings. Importantly, the variance of the ML estimates $\widehat{\mu }$ across the M Monte Carlo trials matches the theoretical sandwich variance, ${\mathsf{\Sigma }}_n^{\mathrm{sand}}=a^2{b}^2/n$. This confirms that the only correct confidence intervals of $\widehat{\mu }$ are those derived from the sandwich estimator.

Table 4. Coverage (in %) of the true mean $\mu =a·b$ of the data-generating process $\mathcal{G}(a,b)$ for the $100(1-\alpha )$% credible intervals of parameter $\widehat{\mu }$ of $\mathcal{E}\left(\mu \right)$ obtained from MCMC simulation (=naive MCMC), OFS-adjusted naive posterior samples, and magnitude-, curvature-, and sandwich-adjusted DREAM_(ZS) algorithms. Results are based on $M={10}^4$ trials with $a=0.5$, $b=0.2$ and $n=100$.

Method	Likelihood	$\mathit{\alpha }=0.01$	$\mathit{\alpha }=0.05$	$\mathit{\alpha }=0.1$	$\mathit{\alpha }=0.2$	$\mathit{\alpha }=0.3$	$\mathit{\alpha }=0.4$	$\mathit{\alpha }=0.5$
		99%	95%	90%	80%	70%	60%	50%
Naive MCMC	$L_n\left(\mu \right)$	92.33	83.11	75.30	63.69	53.38	44.64	36.38
OFS adjustment	$L_n\left(\mu \right)$	94.93	90.69	85.98	76.36	66.57	57.04	47.46
Algorithm 1	$L_n{\left(\mu \right)}^k$	98.39	93.67	88.98	78.61	68.89	58.93	48.92
Algorithm 2	$L_n^{\mathrm{ca}}\left(\mu \right)$	98.41	93.69	88.86	78.87	68.80	59.14	48.86
Algorithm 3	$L_n^{\mathrm{p}}(\mu \mid \lambda )$	98.49	93.78	88.96	78.58	68.96	59.44	49.31

documents the nominal coverage probabilities of the credible regions obtained from MCMC simulation with the DREAM_(ZS) algorithm using the log-likelihood ${\mathcal{L}}_n\left(\mu \right)$ of Equation (A10) (=naive estimator), OFS-adjusted naive posterior samples of Equation (11), magnitude-adjusted log-likelihood $k{\mathcal{L}}_n\left(\mu \right)$ with omnibus scalar k of Equation (10) (=Algorithm 1), curvature-adjusted log-likelihood ${\mathcal{L}}_n^{\mathrm{ca}}\left(\mu \right)={\mathcal{L}}_n\left(\widehat{\mu }+{\mathbf{C}}_n(\mu -\widehat{\mu })\right)$ of Equation (15) (=Algorithm 2), and centralized power log-likelihood ${\mathcal{L}}_n^{\mathrm{p}}(\mu \mid \lambda )$ of Equation (21) (=Algorithm 3).

The curvature-adjustment matrix ${\mathbf{C}}_n$ is a scalar in this case, and according to Equations (18) and (29), we yield ${\mathbf{C}}_n=m_{\omega }{s}_{\omega }^{-1}$, where $s_{\omega }^{-1}$ is the reciprocal of the sample standard deviation of the $\omega$’s. For OFS-adjustment, we substitute the expressions for ${\mathbf{A}}_{*}$ and ${\mathbf{B}}_{*}$ of Equation (A7) into Equation (12) and yield ${\mathsf{\Psi }}_n=s_{\omega }/m_{\omega }$. The tabulated values confirm that

• The asymptotic covariance matrix of the Metropolis algorithm is a single slice of bread. The 100$(1-\alpha )$% credible intervals are in agreement with the frequentist confidence intervals of the naive variance estimator, ${\mathsf{\Sigma }}_n^{\mathrm{naive}}$ in Table 3 and underestimate the theoretic coverage probabilities.
• The OFS adjustment of Equation (11) enlarges the spread of the naive posterior samples but the coverage probabilities of the so-obtained sandwich credible regions underestimate their counterparts of the sandwich estimator in Table 3.
• The three MCMC recipes discussed in this paper successfully join a single slice of bread ${\displaystyle \frac{1}{n}}{\mathbf{A}}_n^{-1}$ to the open-faced sandwich ${\mathbf{B}}_n{\mathbf{A}}_n^{-1}$ to produce the sandwich variance ${\mathsf{\Sigma }}_n^{\mathrm{sand}}$. The coverage probabilities of the 100$(1-\alpha )$% credible regions of Algorithms 1–3 match those of the sandwich estimator in Table 3.
• The tabulated values for Algorithm 3 are the first proof that the centralized power log-likelihood function ${\mathcal{L}}_n^{\mathrm{p}}(\mu \mid \lambda )$ of Equation (21) works in practice. This inspires confidence that we can sample the sandwich distribution without using matrix square roots.

The OFS adjustment is computationally appealing and enlarges the spread of the naive posterior samples, yet the so-obtained credible regions underestimate the theoretical coverage probabilities. Magnitude, curvature, and kernel adjustment of the log-likelihood function all appear viable methods for sandwich-adjusted MCMC simulation. There are important differences between these three sampling methods and the practical consequences of this are better illustrated with a multivariate target distribution.

Having completed the above exercise, we now replace $\mathcal{G}(a,b)$ with alternative distributions for the data-generating process. Figure 5 shows histograms of the omnibus scalar k when the data-generating process is (a) $\mathcal{G}(a,b)$, (b) $\mathcal{N}(\mu ,{\sigma }^2)$, (c) $\mathcal{LOGN}(\mu ,\sigma )$, (d) $\mathcal{W}(\alpha ,\beta )$, and (e) $\mathcal{B}(a,b)$. For comparability, the scale, shape, and/or location parameters of each distribution are chosen such that $\mathbb{E}\left[\omega \right]=1/2$ and $Var\left[\omega \right]=1/10$. The theoretical value of the omnibus scalar for each distribution is $k={\mu }^2{\sigma }^{-2}=2.5$.

The histograms of $\widehat{k}$ appear remarkably similar across the different distributions. This confirms that our inferences for $\mu$ are robust and do not depend on the distribution of the data-generating process. The marginal distributions of the omnibus scalar center on the theoretic value of $k=2.5$ and display a small right tail. The dispersion of $\widehat{k}$ is a result of sample size and will disappear if we set n much larger in the Monte Carlo trials.

We now move on to our third and last case study. This will involve the use of real-world data and a multivariate posterior distribution.

6.3. Case Study 3

Our third and final case study examines the streamflow response of the Leaf River near Collins, MS, USA. The precipitation–discharge transformation is simulated using the Xinanjiang conceptual watershed model originally developed by Zhao and Zhuang [65]. We adopt the implementation of Jayawardena and Zhou [75] and Knoben et al. [76], augmented with a pan evaporation parameter and three linear routing reservoirs. This configuration comprises seven control volumes that conceptually represent water storage and routing. Appendix E provides a detailed description of the Xinanjiang model structure, including the control volumes, state variables, flux relationships, and routing scheme used to convert areal average precipitation into total channel inflow and river discharge. The model equations are solved using a mass-conservative, second-order integration method with adaptive time stepping, ensuring both numerical stability and accuracy. A one-year spin-up period removes the influence of state variable initialization.

Table A2. Description of Xinanjiang parameters, including symbols, units, lower and upper bounds.

Symbol	Description	Units	Min.	Max.
$f_{\mathrm{p}}$	Ratio of potential evapotranspiration to pan evaporation	-	$0.5$	$1.5$
$A_{\mathrm{im}}$	Impervious area	-	${10}^{-4}$	${10}^{-1}$
a	Tension water distribution inflection parameter	-	$-0.5$	$0.5$
b	Tension water distribution shape parameter	-	${10}^{-1}$	2
$f_{\mathrm{wm}}$	Fraction of $s_{\mathrm{tot}}$ that is $w_{\max }$	-	${10}^{-3}$	1
$f_{\mathrm{lm}}$	Fraction of $W_{\max }$ that is tension water threshold for evaporation change	-	${10}^{-3}$	1
c	Fraction of tension water threshold for second evaporation change	-	${10}^{-3}$	1
$s_{\mathrm{tot}}$	Total soil moisture storage	mm	1	${10}^3$
$\beta$	Free water distribution shape parameter	-	${10}^{-3}$	2
$k_{\mathrm{i}}$	Free water interflow parameter	d−1	${10}^{-3}$	3
$k_{\mathrm{g}}$	Free water groundwater parameter	d−1	${10}^{-3}$	1
$c_{\mathrm{i}}$	Interflow time coefficient	d−1	${10}^{-3}$	1
$c_{\mathrm{g}}$	Baseflow time coefficient	d−1	${10}^{-3}$	1
$k_{\mathrm{f}}$	Recession constant of routing reservoirs	d−1	${10}^{-1}$	5

lists the 14 parameters of the Xinanjiang model to be estimated from streamflow measurements. For inference, we express the Xinanjiang model as the vector-valued regression

$$\mathsf{\omega }=\mathbf{f}(\mathsf{\theta },\mathbf{I})+\mathbf{e},$$

where $\mathsf{\omega }={({\omega }_1,\dots ,{\omega }_n)}^{\top }$ is the $n\times 1$ vector of discharge observations, $\mathsf{\theta }={(f_{\mathrm{p}},A_{\mathrm{im}},a,b,f_{\mathrm{wm}},f_{\mathrm{lm}},c,s_{\mathrm{tot}},\beta ,k_{\mathrm{i}},k_{\mathrm{g}},c_{\mathrm{i}},c_{\mathrm{g}},k_{\mathrm{f}})}^{\top }$ signifies the parameter vector, $\mathbf{I}$ is the $n\times 2$ matrix of exogenous variables containing daily areal-average rainfall and potential evapotranspiration, and $\mathbf{e}={(e_1,\dots ,e_n)}^{\top }$ is the $n\times 1$ vector of discharge measurement errors. We assign a uniform prior $P\left(\mathsf{\theta }\right)$ over the bounds given in Table A2 and use the standardized skewed-t (SST) density of Scharnagl et al. [77] to evaluate agreement between observed and simulated streamflows

$$\begin{array}{cc}\hfill f_{\mathrm{SST}}({ϵ}_t\mid 0,1,\nu ,\xi )& ={\displaystyle \frac{2{\sigma }_{\nu \xi }}{(\xi +{\xi }^{-1})}}{\displaystyle \frac{\mathsf{\Gamma }\left((\nu +1)/2\right)}{\mathsf{\Gamma }(\nu /2)\sqrt{\pi (\nu -2)}}}{\left[1+{\displaystyle \frac{1}{\nu -2}}{\left({\displaystyle \frac{{\mu }_{\nu \xi }+{\sigma }_{\nu \xi }{ϵ}_t}{{\xi }^{sign({\mu }_{\nu \xi }+{\sigma }_{\nu \xi }{ϵ}_t)}}}\right)}^2\right]}^{-(\nu +1)/2},\hfill \end{array}$$

(30)

where ${ϵ}_t=e_t/s_t$ is the t^th studentized streamflow residual, $sign\left(x\right)=\left|x\right|/x$, denotes the signum function, and scalars ${\mu }_{\nu \xi }=M_1(\xi -{\xi }^{-1})$ and ${\sigma }_{\nu \xi }={\{(M_2-M_1^2)({\xi }^2+{\xi }^{-2})+2M_1^2-M_2\}}^{1/2}$ are shift and scale constants, respectively, which depend on the degrees of freedom $\nu >2$, the skewness parameter $\xi >0$, and the first and second absolute moments $M_1$ and $M_2$ of the SST density [57,77]. The total likelihood $L_n(\mathsf{\theta },\nu ,\xi )$ for a n-record of studentized residuals ${ϵ}_1\left(\mathsf{\theta }\right),\dots ,{ϵ}_n\left(\mathsf{\theta }\right)$ is now equal to

$$\begin{array}{cc}\hfill L_n(\mathsf{\theta },\nu ,\xi )& =C(\nu ,\xi ,n)\prod _{t=1}^n{\left[1+{\displaystyle \frac{1}{\nu -2}}{\left({\displaystyle \frac{{\mu }_{\nu \xi }+{\sigma }_{\nu \xi }{ϵ}_t\left(\mathsf{\theta }\right)}{{\xi }^{sign({\mu }_{\nu \xi }+{\sigma }_{\nu \xi }{ϵ}_t\left(\mathsf{\theta }\right))}}}\right)}^2\right]}^{-(\nu +1)/2},\hfill \end{array}$$

where the prefactor $C(\nu ,\xi ,n)$ is

$$\begin{array}{cc}\hfill C(\nu ,\xi ,n)& ={\left({\displaystyle \frac{2{\sigma }_{\nu \xi }}{(\xi +{\xi }^{-1})}}{\displaystyle \frac{\mathsf{\Gamma }\left((\nu +1)/2\right)}{\mathsf{\Gamma }(\nu /2)\sqrt{\pi (\nu -2)}}}\right)}^n.\hfill \end{array}$$

The measurement error standard deviation $s_t$ of the t^th streamflow observation ${\omega }_t$ is modeled as a linear function of the simulated discharge $y_t\left(\mathsf{\theta }\right)$ under model parameters $\mathsf{\theta }$

$$\begin{array}{cc}\hfill s_t& =s_0+s_1{y}_t\left(\mathsf{\theta }\right),\hfill \end{array}$$

where the intercept $s_0={10}^{-4}$ (mm/d) is fixed at a small positive value, and the slope $s_1>0$ is determined offline so as to enforce unit variance of the studentized raw residuals ${ϵ}_1\left(\mathsf{\theta }\right),\dots ,{ϵ}_n\left(\mathsf{\theta }\right)$. The slope is obtained via an iterative root-finding procedure described in detail by Vrugt et al. [57]. With this variance model, the Student–t log-likelihood becomes

$$\begin{array}{cc}\hfill {\mathcal{L}}_n^{\mathrm{s}}(\mathsf{\theta },\nu ,\xi \mid s_0)& \simeq nlog\left(C(\nu ,\xi ,1)\right)-\sum _{t=1}^n\{log(|s_0+s_1{y}_t\left(\mathsf{\theta }\right)\left|\right)\}-{\displaystyle \frac{\nu +1}{2}}\sum _{t=1}^n\left\{log\left[1+{\displaystyle \frac{1}{\nu -2}}{\left({\displaystyle \frac{{\mu }_{\nu \xi }+{\sigma }_{\nu \xi }{ϵ}_t\left(\mathsf{\theta }\right)}{{\xi }^{sign({\mu }_{\nu \xi }+{\sigma }_{\nu \xi }{ϵ}_t\left(\mathsf{\theta }\right))}}}\right)}^2\right]\right\}.\hfill \end{array}$$

(31)

To facilitate both pairwise and parameter-wise comparisons of the $d\times d$ sensitivity ${\mathbf{A}}_n$ and variability ${\mathbf{B}}_n$ matrices, we apply the affine rescaling

$$\begin{array}{cc}\hfill {\underline{\theta }}_j& ={\displaystyle \frac{{\theta }_j-{\theta }_j^{\min }}{{\theta }_j^{\max }-{\theta }_j^{\min }}}\mathrm{for}j=1,\dots ,14,\mathrm{and}{\underline{\eta }}_r={\displaystyle \frac{{\eta }_r-{\eta }_r^{\min }}{{\eta }_r^{\max }-{\eta }_r^{\min }}}\mathrm{for}r=1,2,\hfill \end{array}$$

which maps the Xinanjiang parameters $\mathsf{\theta }={({\theta }_1,\dots ,{\theta }_{14})}^{\top }$ and nuisance variables $\mathsf{\eta }={(\nu ,\xi )}^{\top }$ onto the unit hypercube. Inference is then conducted on the normalized parameters

$$\begin{array}{cc}\hfill \underline{\mathsf{\theta }}& ={({\underline{f}}_{\mathrm{p}},{\underline{A}}_{\mathrm{im}},\underline{a},\underline{b},{\underline{f}}_{\mathrm{wm}},{\underline{f}}_{\mathrm{lm}},\underline{c},{\underline{s}}_{\mathrm{tot}},\underline{\beta },{\underline{k}}_{\mathrm{i}},{\underline{k}}_{\mathrm{g}},{\underline{c}}_{\mathrm{i}},{\underline{c}}_{\mathrm{g}},{\underline{k}}_{\mathrm{f}})}^{\top },\hfill \end{array}$$

and normalized nuisance variables $\underline{\mathsf{\eta }}={(\underline{\nu },\underline{\xi })}^{\top }$. Prior to Xinanjiang model execution, $\underline{\mathsf{\theta }}$ is transformed back to the original parameter scales using the lower and upper bounds in Table A2. The prior distributions for the degrees of freedom and skewness parameters are uniform with support $\nu \in (2,{10}^4]$ and $\xi \in [{10}^{-1},{10}^2]$, respectively.

Figure 6 shows histograms of the marginal posterior distributions of the normalized Xinanjiang model parameters obtained using the DREAM_(ZS) algorithm.

The Markov chain sample with highest value of $P_n(\underline{\mathsf{\theta }},\underline{\nu },\underline{\xi })=P(\underline{\mathsf{\theta }},\underline{\nu },\underline{\xi })L_n^{\mathrm{s}}(\underline{\mathsf{\theta }},\underline{\nu },\underline{\xi }\mid s_0={10}^{-4})$ (red square) coincides almost perfectly with the ML solution (red cross) of the frequentist estimator separately obtained by maximizing the Student t likelihood $L_n^{\mathrm{s}}(\underline{\mathsf{\theta }},\underline{\nu },\underline{\xi }\mid s_0={10}^{-4})$ using a gradient-based optimization method. For all Xinanjiang model parameters except the tension water inflection parameter a and the free water shape parameter $\beta$, the MCMC-sampled marginal posterior distributions are unimodal, bell-shaped and centered around the ML solution. In contrast, the marginal posterior distribution of a is approximately uniform on the interval between 0–$0.4$, whereas the density function of $\beta$ has a trapezoidal shape. The MAP values of these two parameters do not coincide with distinct posterior peaks, yet are in close vicinity of their ML estimates.

Most of the MCMC-sampled posterior histograms are in close agreement with the normal marginal distributions (blue lines) derived from the naive variance ${\mathsf{\Sigma }}_n^{\mathrm{naive}}={\displaystyle \frac{1}{n}}{\mathbf{A}}_n^{-1}$ of the frequentist estimator where the sensitivity matrix ${\mathbf{A}}_n=-{\displaystyle \frac{1}{n}}{\nabla }^2{\mathcal{L}}_n^{\mathrm{s}}({\widehat{\underline{\mathsf{\theta }}}}_{*},{\widehat{\underline{\nu }}}_{*},{\widehat{\underline{\xi }}}_{*}\mid s_0={10}^{-4})$ is computed from the second-order partial derivatives of the Student t likelihood function. This frequentist estimator assumes model linearity and a symmetric Gaussian distribution around the ML estimate. In contrast, the MCMC method approximates the marginal distributions of the parameters from a large sample of posterior realizations. This enables MCMC to account for nonlinear model relationships and represent arbitrary posterior shapes, including skewed and heavy-tailed distributions. As a result, frequentist and Bayesian estimates of parameter uncertainty may differ. In the literature, this distinction is often framed in terms of linear versus nonlinear confidence intervals. However, in the Bayesian context, the more appropriate term is credible intervals, which reflect the probabilistic interpretation of uncertainty inherent to Bayesian inference.

Table A3. MCMC-derived naive posterior correlation matrix of the Xinanjiang model parameters and $\nu$ and $\xi$ using the Student t likelihood.

in Appendix F shows that most Xinanjiang parameters exhibit only weak correlations. Notable exceptions are the recession parameters $k_{\mathrm{i}}$ and $k_{\mathrm{g}}$ of the interflow and groundwater reservoirs, respectively, which display a very strong correlation ($r=0.985$), followed by a correlation of $r=0.856$ between the tension water inflection parameter a and the total soil moisture storage $s_{\mathrm{tot}}$, and $r=0.782$ between the fraction of tension water storage $f_{\mathrm{wm}}$ and the free water distribution shape parameter $\beta$. The generally low posterior correlation coefficients account in part for the close agreement between the naive posterior histograms of the Xinanjiang parameters and the normal marginal distributions derived from the frequentist estimator.

Before examining the posterior Xinanjiang parameter distributions obtained from Algorithms 1–3, we first take a closer look in

Table 5. Total sensitivity ${\mathbf{A}}_n^{\mathrm{s}}$ and variability ${\mathsf{\beta }}_n^{\mathrm{s}}$ matrices of the Xinanjiang model parameters and nuisance variables $\nu$ and $\xi$ of the Student t likelihood.

at the bread and meat matrices of the Student t likelihood $L_n^{\mathrm{s}}(\mathsf{\theta },\nu ,\xi \mid s_0={10}^{-4})$. Comparing these two matrices offers insight into the magnitude of the sandwich correction.

The main diagonal entries of ${\mathbf{A}}_n^{\mathrm{s}}$ and $\mathsf{\beta }{n}_{\mathrm{s}}$ are in relatively poor agreement. The bolded entries of $\mathsf{\beta }{n}_{\mathrm{s}}$ are nearly an order of magnitude larger than their counterparts of ${\mathbf{A}}_n^{\mathrm{s}}$. This gives rise to a value of $k=0.1327$ for the omnibus scalar of Pauli et al. [31] in Equation (10). This value is far removed from the desired value of $k=1$ under correct specification. In Section 9, we formulate several other quantitative measures of (dis)similarity between the bread and meat matrices. This includes the Frobenius norm of the naive and sandwich variance matrices in Equation (35). The norm exceeds $2.0$, indicating substantial misspecification and underscoring the need for the sandwich estimator to robustly quantify Xinanjiang parameter uncertainty.

In

Table A4. Frequentist ${\mathbf{A}}_n^{\mathrm{s}}$ and Bayesian ${\widehat{\mathbf{A}}}_n^{\mathrm{s}}$ estimator of the bread matrix of the Xinanjiang model parameters and nuisance variables $\nu$ and $\xi$ for the Student t log-likelihood function.

of Appendix G we compare the frequentist bread matrix ${\mathbf{A}}_n^{\mathrm{s}}$ with the inverse of the covariance matrix of the DREAM_(ZS)-sampled naive posterior realizations. The MCMC-derived bread matrix is in reasonable agreement with ${\mathbf{A}}_n^{\mathrm{s}}$, consistent with the close correspondence observed in Figure 6 between the frequentist characterization of naive parameter uncertainty and the normal posterior histograms sampled by the DREAM_(ZS) algorithm for most parameters. The marginal posterior distributions of the tension water inflection parameter a and the free water shape parameter $\beta$ deviate noticeably from normality, which explains in part the relatively large differences in their diagonal elements of the bread matrices of the frequentist and MCMC methods. The largest discrepancy is observed for the parameter $f_{\mathrm{wm}}$, whose MCMC-derived bread matrix value on the main diagonal is $0.122$, approximately 40 times smaller than the corresponding value of $4.860$ from the frequentist estimator. The culprit may be the prior distribution, which truncates the posterior distribution of $f_{\mathrm{wm}}$ at unity but does not affect the normal approximation underlying the frequentist characterization of naive parameter uncertainty. Thus, in summary, good agreement between the linear (frequentist-based) and nonlinear (Bayesian sample-based) estimates of the sensitivity (bread) matrix suggests that the posterior distribution is approximately Gaussian, and that the multinormal frequentist description of the ML uncertainty is consistent with the fully Bayesian approach.

Figure 7 presents histograms of the OFS-adjusted posterior samples of the Xinanjiang parameters and degrees of freedom $\nu$ of the Student t likelihood function $L_n^{\mathrm{s}}(\mathsf{\theta },\nu ,\xi \mid s_0={10}^{-4})$. The OFS-adjusted posterior samples are derived from Equation (11) using ${\mathsf{\Psi }}_n={\mathbf{A}}_n^{-1}{\mathbf{B}}_n^{1/2}{\mathbf{A}}_n^{1/2}$ where the matrix square roots, ${\mathbf{A}}_n^{1/2}$ and ${\mathbf{B}}_n^{1/2}$, are computed according to Equation (13) using singular value decomposition.

The OFS-adjustment enhances substantially the dispersion of the posterior samples for all Xinanjiang parameters but the tension water inflection parameter a. The histograms of the OFS-adjusted posterior samples (green bars) stretch far beyond the normal marginal distributions (blue lines) derived from the sensitivity matrix ${\mathbf{A}}_n$ of second-order partial derivatives of the Student t log-likelihood function w.r.t. the ML solution $\{\mathsf{\theta },\nu ,\xi \}$ and the naive (blue) histograms of the Xinanjiang parameters. The culprit is model misspecification and, consequently, a poor alignment of the sensitivity ${\mathbf{A}}_n$ and variability ${\mathsf{\beta }}_n$ matrices. The OFS-derived sandwich histograms of the Xinanjiang parameters are in reasonable agreement with the normal marginal distributions (green lines) of the frequentist estimator of ${\mathsf{\Sigma }}_n^{\mathrm{sand}}$. Note that the OFS-adjusted sandwich density functions for $f_{\mathrm{wm}}$, $\beta$, and $k_{\mathrm{i}}$ are visibly lower than their corresponding frequentist densities. This discrepancy arises because the OFS transformation in Equation (11) does not honor the unit interval of the normalized Xinanjiang parameters. Infeasible parameter values lower the probability density of the adjusted posterior samples within the admissible range. Last but not least, for several parameters, the OFS adjustment of Equation (11) altered the location of the mode (peak) of the sandwich distribution. The most notable shifts occurred for $f_{\mathrm{p}}$, b, $f_{\mathrm{lm}}$, c, $k_{\mathrm{g}}$, and $k_{\mathrm{f}}$. Such changes are somewhat counterintuitive and arise in part from the non-uniqueness of the matrix square roots ${\mathbf{A}}_n^{1/2}$ and ${\mathbf{B}}_n^{1/2}$ used in the adjustment.

Figure 8 presents a matrix plot of the bivariate 95% confidence (lines) and credible (dots) regions of all pairs of Xinanjiang parameters. The blue area corresponds to the naive variance whereas the green area is associated with the sandwich-adjusted posterior samples of Algorithm 3.

The bivariate scatter plots offer a clearer depiction of the naive and sandwich uncertainty estimates for the Xinanjiang parameters. The following conclusions can be drawn.

• The naive Bayesian 95% credible regions (blue squares), as sampled by the DREAM_(ZS) algorithm are in strong agreement with the frequentist 95% confidence ellipsoids derived from the naive variance estimator. There are some notable exceptions, particularly in the bivariate scatter plots involving parameter a, where the MCMC-sampled naive confidence regions exceed the frequentist ellipsoids. This is a well-known phenomenon that highlights the distinction between linear and nonlinear confidence (or credible) regions [49,78,79,80].
• The 95% credible regions of the sandwich-adjusted posterior samples (green dots) extend well beyond the sandwich ellipsoids (green lines) of the frequentist estimator. These linear sandwich confidence regions substantially underestimate the true parameter uncertainty, and appear woefully inadequate for accurately characterizing Xinanjiang discharge uncertainty.
• For most parameter pairs, the MCMC-derived sandwich credible regions are unimodal and well described by a bivariate normal distribution.
• The sandwich credible regions of the Xinanjiang parameters are much larger than their naive counterparts. This is a result of misspecification and confirms that the sensitivity (bread) matrix ${\mathbf{A}}_n^{\mathrm{s}}$ of the Student t likelihood function substantially overestimates the information content of the discharge observations. The only valid currency of discharge data informativeness under model misspecification is the Godambe information, as expressed by the sandwich credible regions. The enlarged parameter uncertainty should yield the appropriate parameter coverage probabilities.

Substantial differences between linear and nonlinear confidence regions, such as those observed for the tension water inflection parameter a, often signal problems in model formulation. Other indicators of model misspecification include parameters whose MAP estimates occur at or near the bounds of their prior ranges. Although the Xinanjiang model does not exhibit this behavior for the Leaf River dataset, practical experience with other conceptual hydrologic models suggests that such issues are more common than rare. When a MAP estimate lies close to a parameter bound, the local curvature of the log-likelihood becomes poorly defined, making it difficult or impossible to compute a stable Hessian (bread) matrix. This, in turn, undermines the validity of asymptotic approximations in frequentist inference, such as the ML sandwich estimator used herein.

Before turning our attention to Xinanjiang discharge uncertainty, we first examine in Figure 9 bivariate scatter plots of the OFS-adjusted posterior samples and their counterparts obtained from magnitude-, curvature-, and sandwich-adjusted MCMC simulation. For a direct comparison of the different methods, the same x- and y-axis limits are used for all four graphs in each column. We focus our attention on only a subset of the Xinanjiang parameter pairs.

The results in Figure 9 highlight several interesting observations.

• The sandwich credible regions for the Xinanjiang parameters vary substantially across different sandwich-adjustment methods and often diverge from the ellipsoidal confidence regions obtained using the frequentist sandwich estimator.
• The OFS-adjusted posterior samples in the top panel yield, on average, the smallest 95% credible regions for the Xinanjiang parameters. These regions are straightforward to construct from the naive posterior samples but systematically underestimate the width of the frequentist sandwich confidence regions (green lines). Moreover, the OFS transformation of Equation (11) does not guarantee preservation of the posterior mode. This is evident in Figure 7, where the peak of the OFS-adjusted posterior distributions has shifted away from the ML/MAP solution.
• Magnitude-adjusted MCMC simulation yields, on average, the largest credible regions for the Xinanjiang parameters. The sandwich credible regions of this method usually extend beyond the frequentist sandwich ellipsoids, although not necessarily in both directions of parameter space. The magnitude-adjusted sandwich uncertainty is particularly large for the parameter pairs $a-b$ and $k_{\mathrm{i}}-b$, as shown in Figure 9b2 and Figure 9f2, respectively, with credible regions that extend almost the entire parameter space and appear truncated by the boundaries of the uniform prior distribution. This behavior may be an artifact of the omnibus scalar k, which does not preserve the directional asymmetries inherent in the bread and meat matrices. This requires the use of a separate scaling factor for each Xinanjiang parameter.
• The 95% credible regions derived from curvature-adjusted MCMC simulation are the overall closest match to the 95% sandwich ellipsoids obtained from the frequentist estimator. The prime examples are the credible regions of $f_{\mathrm{p}}-A_{\mathrm{im}}$ (Figure 9a3), $a-b$ (Figure 9b3), $f_{\mathrm{lm}}-c$ (Figure 9d3), and $c_{\mathrm{g}}-k_{\mathrm{f}}$ (Figure 9h3). The sandwich credible regions center on the ML estimator, have a single peak, and appear well described by a multinormal distribution. Large discrepancies between the frequentist confidence regions and curvature-adjusted credible regions are visible for $f_{\mathrm{wm}}-f_{\mathrm{lm}}$ and $k_{\mathrm{g}}-c_{\mathrm{i}}$ in Figure 9c3 and Figure 9g3, respectively.
• The sandwich-adjusted credible regions obtained from our SAMH algorithm closely align with those derived from curvature-adjusted MCMC simulation, though with a slightly enlarged dispersion. The sandwich credible regions are a compromise between the results of magnitude- and curvature-adjusted MCMC simulation. This result inspires confidence in the centralized power likelihood $L_n^{\mathrm{p}}(\mathsf{\theta }\mid \lambda )$ of Equation (23) and couples with the dynamic learning rate $\lambda \left(\mathsf{\theta }\right)$ of Equation (25) to successfully infer the sandwich posterior distribution. This method avoids the need for principal matrix square roots ${\mathbf{A}}_n^{1/2}$ and ${\mathbf{B}}_n^{1/2}$ in constructing a Bayesian approximation to the frequentist sandwich distribution. The dynamic learning rate $\lambda \left(\mathsf{\theta }\right)$ redistributes the posterior probability mass away from the ML solution according to the more robust sandwich description of the parameters of the Xinanjiang model.

We cannot prove that the sandwich credible regions are more accurate, as the pseudo-true values of the Xinanjiang model parameters that generated the observed discharge record are unknown. Instead, we rely on statistical theory, which establishes the sandwich estimator as the only valid asymptotic descriptor of data informativeness, and, consequently, parameter uncertainty under model misspecification.

Figure 10 shows posterior predictive bands for simulated streamflow from Xinanjiang over a representative segment of the six-year training period, obtained by propagating posterior draws of $\mathsf{\theta }$ through the model.

The sandwich variance estimator substantially widens the parameter uncertainty induced intervals for simulated streamflow from the Xinanjiang model, as evident in the right-hand panels. The 99% intervals expand markedly, especially near peak flows. Quantitatively, the 99%, 95%, 90% and 68% streamflow intervals based on sandwich parameter uncertainty contain $37.0$%, $26.3$%, $21.1$% and $12.9$% of the discharge observations, respectively, compared with $13.3$%, $10.1$%, $8.7$% and $5.4$% under the naive variance estimator. Thus, the naive 99% intervals achieve roughly the same coverage (≈13%) as the sandwich 68% intervals. In terms of width, the sandwich intervals are about twice as wide at low flows and roughly three times as wide at the hydrograph peaks (see Figure A2 in Appendix H).

Finally, we examine the discharge residuals obtained from the ML parameter values of the Xinanjiang model. Figure 11 shows a histogram of the studentized discharge residuals ${ϵ}_1\left({\widehat{\mathsf{\theta }}}_{*}\right),\dots ,{ϵ}_n\left({\widehat{\mathsf{\theta }}}_{*}\right)$, with gray bars normalized to represent a probability density estimate such that the total area under the bars amounts to one. We also plot the SST density $f_{\mathrm{SST}}(ϵ\mid 0,1,\nu ,\xi )$ of Equation (30) using the ML values of $\nu$ and $\xi$.

The histogram of the discharge residuals is in excellent agreement with the SST density. The studentized residuals ${ϵ}_1\left(\widehat{\mathsf{\theta }}\right),\dots ,{ϵ}_n\left(\widehat{\mathsf{\theta }}\right)$ follow a Student t distribution with $\widehat{\nu }=2.92$ degrees of freedom and skewness $\widehat{\xi }=2.09$. This degrees of freedom is much smaller than one would expect from the sample size $n=1827$ and the number of parameters $p=16$ alone. This result once again confirms that the discharge residuals follow a Laplacian or double exponential distribution [11,57]. A skewness of $\widehat{\xi }=2.09$ indicates that the distribution of MAP discharge residuals is right-skewed. Consequently, the mode (peak) of the distribution of the studentized streamflow residuals is located at $-0.64$, to the left of the median value of $-0.20$, which itself is smaller than the mean studentized residual of approximately $0.054$. This mean value points to a negative bias in the Xinanjiang model, indicating a tendency, on average, to underestimate measured streamflows. The magnitude of this bias is around $-0.14$ mm/d or $11.3$% of the mean measured discharge of $1.25$ mm/d.

The SST density with low degrees of freedom exhibits both a sharper peak near its mean and heavier tails compared to the normal distribution (dotted line). This makes the Student t likelihood more robust to outliers and is well suited for inverse modeling of discharge data with the occasional large streamflow residuals. The largest residuals are typically attributable to precipitation measurement errors and are less governed by structural limitations and/or deficiencies of the hydrologic model. However, the sandwich estimator cannot distinguish between these two error sources. Both count as misspecification.

7. Numerical Estimation of the Sensitivity (Bread) Matrix

The naive and sandwich variance estimators rely on knowledge of the sensitivity matrix ${\mathbf{A}}_n$ and variability matrix ${\mathbf{B}}_n$ both evaluated at the ML estimator ${\widehat{\mathsf{\theta }}}_{*}$. Matrix ${\mathbf{B}}_n={\displaystyle \frac{1}{n}}{\sum }_{i=1}^n\nabla {\mathcal{L}}_{{\omega }_i}\left({\widehat{\mathsf{\theta }}}_{*}\right){\nabla }^{\top }{\mathcal{L}}_{{\omega }_i}\left({\widehat{\mathsf{\theta }}}_{*}\right)$ is constructed solely from first-order derivatives of the log-likelihood. When computed carefully either analytically or through numerical differentiation the resulting matrix is typically symmetric and positive definite. This is not true for the sensitivity matrix ${\mathbf{A}}_n=-{\displaystyle \frac{1}{n}}{\nabla }^2{\mathcal{L}}_n\left({\widehat{\mathsf{\theta }}}_{*}\right)$. The main challenge arises from the second-order derivatives of the log-likelihood, which are more difficult to compute than their first-order counterparts, especially when one or more parameters lie near their lower or upper bounds. The sensitivity matrix ${\mathbf{A}}_n$ must be positive definite and therefore invertible to compute both the naive and sandwich variance estimators. If ${\mathbf{A}}_n$ is not invertible, most textbooks advise that the model should be reconsidered, re-specified, and the analysis rerun, or, in some cases, that additional data should be collected. Holding certain model parameters constant at known or hypothesized values can restore invertibility, but this comes at the cost of a reduced model flexibility and potentially introducing bias if the fixed values are incorrect. Furthermore, model simplification affects the estimates of the remaining variables and therefore the interpretation of the findings [81].

Gill and King [41] suggest using a pseudo-factorization ${\mathbf{A}}_n={\mathbf{V}}^{\top }\mathbf{V}$ of the sensitivity matrix if ${\mathbf{A}}_n$ is not positive definite. Their so-called generalized Cholesky decomposition $\mathbf{V}=\mathrm{gchol}\left({\mathbf{A}}_n\right)$ avoids the failures of earlier factorization methods by Gill and Murray [40] and Gill et al. [82] by selectively modifying small or negative pivots. This yields a controlled decomposition even when ${\mathbf{A}}_n$ is indefinite or nearly singular. The resulting pseudo-variance matrix ${\left({\mathbf{V}}^{\top }\mathbf{V}\right)}^{-1}={\mathbf{V}}^{-1}{\left({\mathbf{V}}^{\top }\right)}^{-1}$ serves as a stand-in for ${\mathbf{A}}_n^{-1}$. While this provides a computational workaround, it does not resolve the underlying invertibility problem; it merely allows variance estimation to proceed despite numerical artifacts. Consequently, when the sensitivity matrix is not invertible, results should be interpreted with caution, and model diagnostics and a careful reevaluation of assumptions remain essential.

Another possibility that requires almost no additional computation is to derive matrix ${\mathbf{A}}_n$ from samples of the naive posterior distribution [25]. Theory establishes that this distribution will be asymptotically normal around the MAP estimator ${\widehat{\mathsf{\theta }}}_{*}$ and with a covariance matrix equal to a single slice of bread ${\displaystyle \frac{1}{n}}{\mathbf{A}}_n^{-1}$. Thus, we can use the post-burn-in naive posterior samples as estimators of the bread matrix, ${\widehat{\mathbf{A}}}_n={\displaystyle \frac{1}{n}}Cov{\left[\{{\mathsf{\theta }}_{\left(b\right)},{\mathsf{\theta }}_{(b+1)},\dots ,{\mathsf{\theta }}_{\left(T\right)}\}\right]}^{-1}$. Alternatively, we retain the results of the evaluations of ${\mathcal{L}}_n\left(\mathsf{\theta }\right)$ at each iteration of the sampler and use them to numerically estimate the Hessian matrix at ${\widehat{\mathsf{\theta }}}_{*}$. This Hessian approximation will generally be a good estimator of ${\mathbf{A}}_n$.

8. Limitations of Sandwich-Adjusted MCMC Simulation

Unlike existing approaches that rely on arbitrary matrix square roots, eigendecompositions or a single scaling factor applied uniformly across the parameter space, our method employs a parameter-dependent learning rate $\lambda \left(\mathsf{\theta }\right)$ that enables direction-specific tempering of the likelihood. This allows the sampler to capture directional asymmetries in the sandwich distribution, particularly under model misspecification or in small-sample regimes, and yields credible regions that remain valid when standard Bayesian inference underestimates uncertainty. In our research for this paper, we identified one potential weakness of our methodology. When the posterior distribution is multimodal and these modes are disconnected, then the learning rate $\lambda \left(\mathsf{\theta }\right)$ can suppress one of the peaks, thereby inflating the probability mass of one or more other peaks. The sandwich-adjusted chains then concentrate on the other modes. Through our investigations, we found that a simple and effective remedy is to restrict the learning rate to the interval $(0,1]$. This preserves the multimodal structure of the posterior sandwich distribution.

9. Formal Measures for the Degree of Model Misspecification

The misalignment of the naive and sandwich variance estimators can be summarized by scalar measures of model misspecification. This idea is not new. For example, White [83] developed an information matrix test to assess whether the discrepancy between ${\mathbf{A}}_n$ and ${\mathbf{B}}_n$ is statistically significant. This is a Wald-type ${\chi }^2$ test: under regularity conditions and correct specification, the stacked elements $\sqrt{n}({\mathbf{B}}_n-{\mathbf{A}}_n)$ are asymptotically jointly normal with mean zero, so the associated quadratic form converges to ${\chi }_{d(d+1)/2}^2$ [84]. This section introduces additional misspecification metrics and presents an information-theoretic interpretation of the misalignment score of Vrugt et al. [8]. These measures complement commonly used model evaluation techniques such as residual diagnostics, which assess the validity of likelihood assumptions about variance, distributional form, and dependence structure [11,57]. In contrast, our proposed metrics do not rely explicitly on residual behavior or associated goodness-of-fit statistics. Instead, they assess misspecification implicitly through structural features, specifically, the alignment between the sensitivity and variability matrices, ${\mathbf{A}}_n$ and ${\mathbf{B}}_n$, which reflect both the model’s internal dynamics and its interaction with the data. These diagnostics help guide model selection and improvement, and serve as a safeguard against overconfidence in model-based inference, particularly in applications where structural model error is difficult to detect or eliminate through residual analysis alone.

In theory, the proposed metrics can be evaluated at any $\mathsf{\theta }\in \mathsf{\Theta }$ provided that the local sensitivity matrix is nonsingular (ideally positive definite) and the local variability matrix is positive semidefinite. In practice, we presuppose calibration and report the metrics at the MAP estimate ${\widehat{\mathsf{\theta }}}_{*}$, using the naive ${\mathsf{\Sigma }}_n^{\mathrm{naive}}$ and sandwich ${\mathsf{\Sigma }}_n^{\mathrm{sand}}$ variances computed from ${\mathbf{A}}_n$ and ${\mathbf{B}}_n$. Without calibration, these quantities are not as meaningful.

9.1. Relative Entropy

Let $F={\mathcal{N}}_d({\widehat{\mathsf{\theta }}}_{*},{\mathsf{\Sigma }}_n^{\mathrm{sand}})$ and $P={\mathcal{N}}_d({\widehat{\mathsf{\theta }}}_{*},{\mathsf{\Sigma }}_n^{\mathrm{naive}})$ denote the d-variate normal distributions of the sandwich and naive variance estimators, respectively. The Kullback and Leibler [19] divergence $d_{\mathrm{KL}}(P,F)$ of F and P equals (derivation in Appendix B of Vrugt [85])

$$\begin{array}{cc}\hfill d_{\mathrm{KL}}\left({\mathcal{N}}_d({\widehat{\mathsf{\theta }}}_{*},{\mathsf{\Sigma }}_n^{\mathrm{naive}}),{\mathcal{N}}_d({\widehat{\mathsf{\theta }}}_{*},{\mathsf{\Sigma }}_n^{\mathrm{sand}})\right)& ={\displaystyle \frac{1}{2}}\left[log\left(\left|{\left({\mathsf{\Sigma }}_n^{\mathrm{naive}}\right)}^{-1}{\mathsf{\Sigma }}_n^{\mathrm{sand}}\right|\right)+tr\left({\left({\mathsf{\Sigma }}_n^{\mathrm{sand}}\right)}^{-1}{\mathsf{\Sigma }}_n^{\mathrm{naive}}\right)-d\right].\hfill \end{array}$$

This statistical distance between the sandwich and naive posterior distributions is also known as the relative entropy from P to F, and equals the multivariate divergence score proposed by Dawid and Sebastiani [56] for identical means. We can admit the bread and meat matrices

$$\begin{array}{cc}\hfill d_{\mathrm{xx}}\left({\mathcal{N}}_d({\widehat{\mathsf{\theta }}}_{*},{\mathsf{\Sigma }}_n^{\mathrm{naive}}),{\mathcal{N}}_d({\widehat{\mathsf{\theta }}}_{*},{\mathsf{\Sigma }}_n^{\mathrm{sand}})\right)& ={\displaystyle \frac{1}{2}}\left[log\left(\left|n{\mathbf{A}}_n{\displaystyle \frac{1}{n}}{\mathbf{A}}_n^{-1}{\mathsf{\beta }}_n{\mathbf{A}}_n^{-1}\right|\right)+tr\left(n{\mathbf{A}}_n{\mathsf{\beta }}_n^{-1}{\mathbf{A}}_n{\displaystyle \frac{1}{n}}{\mathbf{A}}_n^{-1}\right)-d\right]\hfill \\ \hfill & ={\displaystyle \frac{1}{2}}log\left(\right|{\mathsf{\beta }}_n{\mathbf{A}}_n^{-1}\left|\right)+{\displaystyle \frac{1}{2}}tr\left({\mathbf{A}}_n{\mathsf{\beta }}_n^{-1}\right)-{\displaystyle \frac{d}{2}}.\hfill \end{array}$$

(32)

This divergence score is strictly proper, meaning that $d_{\mathrm{xx}}(P,F)$ is nonnegative and zero only if $P=F$, thus, ${\mathsf{\Sigma }}_n^{\mathrm{naive}}={\mathsf{\Sigma }}_n^{\mathrm{sand}}$. The greater the misalignment between the sensitivity and variability matrices, the larger the value of $d_{\mathrm{xx}}(P,F)$. The subscript ‘xx’ is intentionally used as a neutral placeholder, and we leave the formal naming of this divergence to future users or the broader research community. The misalignment score of Equation (32) is particularly well suited for applications in machine learning, where the sensitivity (bread) matrix, ${\mathbf{A}}_n$ and variability (meat) matrix ${\mathsf{\beta }}_n$ can often be obtained “for free” as by-products of automatic differentiation. The misalignment score satisfies $d_{\mathrm{xx}}(P,F)=\mathbb{H}(P,F)-\mathbb{H}\left(P\right)$, where

$$\begin{array}{cc}\hfill \mathbb{H}(P,F)& ={\displaystyle \frac{1}{2}}log\left({\left(2\pi \right)}^d\left|{\displaystyle \frac{1}{n}}{\mathbf{A}}_n^{-1}{\mathsf{\beta }}_n{\mathbf{A}}_n^{-1}\right|\right)+{\displaystyle \frac{1}{2}}tr\left(n{\mathbf{A}}_n{\mathsf{\beta }}_n^{-1}{\mathbf{A}}_n{\displaystyle \frac{1}{n}}{\mathbf{A}}_n^{-1}\right)\left(\left|{\displaystyle \frac{1}{n}}{\mathbf{A}}_n^{-1}{\mathsf{\beta }}_n{\mathbf{A}}_n^{-1}\right|=n^{-d}|{\mathbf{A}}_n^{-1}{|}^2\left|{\mathsf{\beta }}_n\right|\right)\hfill \\ \hfill & ={\displaystyle \frac{d}{2}}log\left(2\pi \right)-{\displaystyle \frac{d}{2}}log\left(n\right)-log\left(\right|{\mathbf{A}}_n\left|\right)+{\displaystyle \frac{1}{2}}log\left(\right|{\mathsf{\beta }}_n\left|\right)+{\displaystyle \frac{1}{2}}tr\left({\mathbf{A}}_n{\mathsf{\beta }}_n^{-1}\right),\hfill \end{array}$$

is the cross-entropy between the d-variate normal naive and sandwich distributions and

$$\begin{array}{cc}\hfill \mathbb{H}\left(P\right)& ={\displaystyle \frac{1}{2}}log\left({\left(2\pi e\right)}^d\left|{\displaystyle \frac{1}{n}}{\mathbf{A}}_n^{-1}\right|\right)\hfill \\ \hfill & ={\displaystyle \frac{d}{2}}+{\displaystyle \frac{d}{2}}log\left(2\pi \right)-{\displaystyle \frac{d}{2}}log\left(n\right)-{\displaystyle \frac{1}{2}}log\left(\right|{\mathbf{A}}_n\left|\right),\hfill \end{array}$$

is the differential entropy of the multinormal naive distribution. This formulation highlights the misalignment score’s role as an information-theoretic measure of misspecification and closes the circle with our earlier work on probabilistic model evaluation [85].

The misalignment score of Equation (32) can be directly compared across models with differing numbers of parameters. It should yield the same model ranking as the logarithmic score, or expected log predictive density

$$S_{\mathrm{LS}}(\mathbb{P},\omega )={\displaystyle \frac{1}{n}}\sum _{t=1}^{n}log\left(f_{P_t}({\omega }_t\mid \mathcal{M})\right)$$

(33)

where $\mathbb{P}=\{P_1,\dots ,P_n\}$ are the posterior predictive distributions under model $\mathcal{M}$ for the naive estimator. This forms the basis of model selection criteria such as the widely applicable information criterion or WAIC and leave-one-out cross-validation [86,87]. If so desired, the misalignment score can be normalized to explicitly account for the number of model parameters d. This yields a per-dimension misalignment score ${\overline{d}}_{\mathrm{xx}}(P,F)$

$$\begin{array}{cc}\hfill {\overline{d}}_{\mathrm{xx}}\left({\mathcal{N}}_d({\widehat{\mathsf{\theta }}}_{*},{\mathsf{\Sigma }}_n^{\mathrm{naive}}),{\mathcal{N}}_d({\widehat{\mathsf{\theta }}}_{*},{\mathsf{\Sigma }}_n^{\mathrm{sand}})\right)& ={\displaystyle \frac{1}{2}}{d}^{-1}log\left(\right|{\mathsf{\beta }}_n{\mathbf{A}}_n^{-1}\left|\right)+{\displaystyle \frac{1}{2}}{d}^{-1}tr\left({\mathbf{A}}_n{\mathsf{\beta }}_n^{-1}\right)-{\displaystyle \frac{1}{2}},\hfill \end{array}$$

(34)

which can be compared across an ensemble of candidate models with differing number of parameters.

9.2. Fréchet Distance

The misalignment between $F={\mathcal{N}}_d({\widehat{\mathsf{\theta }}}_{*},{\mathsf{\Sigma }}_n^{\mathrm{sand}})$ and $P={\mathcal{N}}_d({\widehat{\mathsf{\theta }}}_{*},{\mathsf{\Sigma }}_n^{\mathrm{naive}})$ can also be quantified by the Earth Mover’s or Fréchet distance [88,89]

$$\begin{array}{cc}\hfill d_{\mathrm{F}}\left({\mathcal{N}}_d({\widehat{\mathsf{\theta }}}_{*},{\mathsf{\Sigma }}_n^{\mathrm{naive}}),{\mathcal{N}}_d({\widehat{\mathsf{\theta }}}_{*},{\mathsf{\Sigma }}_n^{\mathrm{sand}})\right)& =\sqrt{tr\left({\mathsf{\Sigma }}_n^{\mathrm{naive}}+{\mathsf{\Sigma }}_n^{\mathrm{sand}}-2{\left({\mathsf{\Sigma }}_n^{\mathrm{naive}}{\mathsf{\Sigma }}_n^{\mathrm{sand}}\right)}^{1/2}\right)}\hfill \\ \hfill & =n^{-1/2}\sqrt{tr\left({\mathbf{A}}_n^{-1}+{\mathbf{A}}_n^{-1}{\mathsf{\beta }}_n{\mathbf{A}}_n^{-1}-2{\left({\mathbf{A}}_n^{-2}{\mathsf{\beta }}_n{\mathbf{A}}_n^{-1}\right)}^{1/2}\right)}.\hfill \end{array}$$

This distance is widely used in machine learning to compare the distribution of generated images from a model against the distribution of real images. Smaller values indicate greater similarity between distributions, with $d_{\mathrm{F}}=0$ corresponding to perfect agreement.

9.3. Frobenius Norm

An alternative diagnostic metric is the Frobenius norm of the difference between the naive and sandwich variance estimators

$$\begin{array}{cc}\hfill {∥P-F∥}_{\mathrm{F}}& =\sqrt{{\sum }_{i=1}^d{\sum }_{j=1}^d{\left({\mathsf{\Sigma }}_{n,ij}^{\mathrm{naive}}-{\mathsf{\Sigma }}_{n,ij}^{\mathrm{sand}}\right)}^2}={\displaystyle \frac{1}{n}}{∥{\mathbf{A}}_n^{-1}({\mathbf{I}}_d-{\mathsf{\beta }}_n{\mathbf{A}}_n^{-1})∥}_{\mathrm{F}},\hfill \end{array}$$

(35)

where smaller values indicate better model specification, and a value of zero is ideal. Larger values imply greater degrees of misspecification. Alternatively, one can compare the observed Fisher ${\widehat{\mathcal{I}}}_n$ and Godambe ${\widehat{\mathcal{G}}}_n$ information matrices at the MAP parameter values

$$\begin{array}{cc}\hfill & {∥{\widehat{\mathcal{I}}}_n-{\widehat{\mathcal{G}}}_n∥}_{\mathrm{F}}=n{∥{\mathbf{A}}_n({\mathbf{I}}_d-{\mathsf{\beta }}_n^{-1}{\mathbf{A}}_n)∥}_{\mathrm{F}}.\hfill \end{array}$$

This yields qualitatively similar conclusions but on a different scale. Further examination of the Fisher-Godambe discrepancy offers valuable insight into the nature and extent of model misspecification, particularly under different modeling assumptions and data sets.

9.4. Herfindahl Index

Under correct specification, the theoretical precision matrix $\mathbf{M}\equiv {\mathbf{A}}_{*}^{1/2\phantom{-1}}{\mathbf{B}}_{*}^{-1\phantom{1/2}}{\mathbf{A}}_{*}^{1/2\phantom{-1}}$ defined in Remark 8 on Page 17 will equal an identity matrix ${\mathbf{I}}_d$. Then, the eigenvalues ${\underline{\lambda }}_1,\dots ,{\underline{\lambda }}_d$ of $\mathbf{M}={\mathbf{I}}_d$ will equal one. Suppose we normalize the d eigenvalues of $\mathbf{M}$

$${\underline{\lambda }}_{i,\mathrm{n}}={\displaystyle \frac{{\underline{\lambda }}_i}{{\sum }_{j=1}^d{\underline{\lambda }}_j}},i=1,\dots ,d,$$

then the Herfindahl index H

$$H=\sum _{i=1}^d{\underline{\lambda }}_{i,\mathrm{n}}^2.$$

is a measure of how dispersed or concentrated the eigenvalues are across the parameter space. This metric is commonly used in economics as a scalar summary of the variance concentration in the principal components of a covariance matrix [90,91]. Under correct specification, all normalized eigenvalues attain a value of $d^{-1}$ and $H=1/d$. This is the lowest possible value for the Herfindahl index and indicates maximum variance uniformity across dimensions. The maximum value $H=1$ is reached when all variability is concentrated in a single direction. Higher Herfindahl indices, thus, imply that most of the uncertainty is concentrated along a few dimensions of the parameter space, potentially indicating ill-conditioning or overfitting. Large differences between the Herfindahl indices of ${\mathsf{\Sigma }}_n^{\mathrm{naive}}$ and ${\mathsf{\Sigma }}_n^{\mathrm{sand}}$ signal model misspecification. In particular, the naive estimator may imply uniformly distributed uncertainty, whereas the sandwich estimator captures the anisotropic structure introduced by model error.

For the identity matrix ${\mathbf{I}}_d$ and diagonal matrix $diag({\underline{\lambda }}_1,\dots ,{\underline{\lambda }}_d)$ we find

$$d_{\mathrm{KL}}\left({\mathcal{N}}_d(\mathbf{0},{\mathbf{I}}_d),{\mathcal{N}}_d\right(\mathbf{0},diag({\underline{\lambda }}_1,\dots ,{\underline{\lambda }}_d)))={\displaystyle \frac{1}{2}}\sum _{i=1}^d\left\{{\underline{\lambda }}_i^{-1}-1+log\left({\underline{\lambda }}_i\right)\right\},$$

and the reverse KL-divergence

$$d_{\mathrm{KL}}({\mathcal{N}}_d\left(\mathbf{0},({\underline{\lambda }}_1,\dots ,{\underline{\lambda }}_d)\right),{\mathcal{N}}_d(\mathbf{0},{\mathbf{I}}_d))={\displaystyle \frac{1}{2}}\sum _{i=1}^d\left\{{\underline{\lambda }}_i-1-log\left({\underline{\lambda }}_i\right)\right\}.$$

The symmetrized KL divergence is ${\displaystyle \frac{1}{4}}{\sum }_{i=1}^d\left\{{\underline{\lambda }}_i-1+{\underline{\lambda }}_i^{-1}\right\}$.

Thus, the Herfindahl index adds to the suite of diagnostics by providing an interpretable scalar summary of the effective dimensionality of the parameter uncertainty. This makes it particularly useful for comparing models of varying complexity or visualizing behavior along a complexity–regularization trade-off. A related measure is the sample variance of ${\underline{\lambda }}_i$’s or $s_{\underline{\lambda }}^2=({\sum }_{i=1}^d{\underline{\lambda }}_i^2-d{m}_{\underline{\lambda }}^2)/(d-1)$, where $m_{\underline{\lambda }}$ is the sample mean of ${\underline{\lambda }}_1,\dots ,{\underline{\lambda }}_d$.

The misspecification diagnostics introduced here serve as a companion to predictive model selection criteria such as the Akaike information criterion (AIC; [92]), the Bayesian information criterion (BIC; [93]), and the WAIC [86]. Whereas these criteria rank models by expected predictive performance under a correctly specified likelihood, our measures assess whether that assumption is credible by quantifying the alignment between the sensitivity and variability matrices. Low misalignment scores support the use of AIC/BIC/WAIC with greater confidence. Large discrepancies warn that their penalties may understate uncertainty and yield overconfident rankings. In practice, the proposed diagnostics can be used to screen out poorly specified models before comparing predictive performance.

In this study, we applied the proposed sandwich-adjusted MCMC simulation method to a collection of discharge data sets and a suite of hydrologic models of varying complexity. For each case, we computed the omnibus scalar k introduced by Pauli et al. [31]. In nearly all applications, the estimated k deviated markedly from the value of unity expected under correct model specification, indicating substantial misspecification across all models. Although more complex models with larger parameter dimensionality d often yielded higher omnibus values, the relationship was not strictly monotonic. This suggests that model structure rather than dimensionality alone plays a critical role in determining specification quality. These findings highlight the practical value of our method for assessing model adequacy and underscore the need for further research on the interplay between model complexity, misalignment scores, other misspecification-based diagnostics, and commonly used residual-based measures of predictive performance.

10. Summary and Conclusions

Frequentist and Bayesian methods are widely used for standard tasks such as statistical inference and hypothesis testing, as well as for more specific tasks including model training (calibration) and prediction (forecasting). In an earlier article, we demonstrated a critical flaw in both maximum likelihood (ML) and Bayesian approaches under model misspecification. Contrary to common teaching and statistical practice, the asymptotic covariance matrix of the ML parameter estimates, ${\mathsf{\Sigma }}_n$, does not equal the inverse of the observed Fisher information matrix, ${\widehat{\mathcal{I}}}_n$. Instead, it corresponds to the sandwich variance matrix ${\mathsf{\Sigma }}_n^{\mathrm{sand}}=\widehat{\mathcal{G}}{\phantom{\underline{\widehat{\mathcal{G}}}}}_n^{-1}$ where the observed Godambe information is defined as ${\widehat{\mathcal{G}}}_n=n{\mathbf{A}}_n^{-1}{\mathbf{B}}_n^{\phantom{-1}}{\mathbf{A}}_n^{-1}$. This Godambe matrix serves as the fundamental measure of data informativeness under model misspecification [8]. Here, ${\mathbf{A}}_n$ and ${\mathbf{B}}_n$ are sample averages of the sensitivity and variability matrices, respectively, for n data points ${\omega }_1,\dots ,{\omega }_n$, evaluated at the ML parameter estimates ${\widehat{\mathsf{\theta }}}_{*}$.

The goals of this paper were three-fold. First, we reviewed and examined three existing methods for producing asymptotically valid sandwich posterior distributions. The first method, known as the open-faced sandwich (OFS) adjustment of Shaby [25], applies direction-specific dilations along the principal axes of the samples of the naive posterior distribution, ${\mathsf{\Sigma }}_n^{\mathrm{naive}}=\widehat{\mathcal{I}}{\phantom{\underline{\widehat{\mathcal{I}}}}}_n^{-1}$ to align its local curvature around the MAP estimator ${\widehat{\mathsf{\theta }}}_{*}$ with that of the sandwich variance matrix ${\mathsf{\Sigma }}_n^{\mathrm{sand}}$. Specifically, naive posterior samples ${\mathsf{\theta }}_{\left(1\right)},\dots ,{\mathsf{\theta }}_{\left(M\right)}$ are centered on the MAP estimator and pre-multiplied by the matrix ${\mathsf{\Psi }}_n={\mathbf{A}}_n^{-1\phantom{1/2}}{\mathbf{B}}_n^{1/2}{\mathbf{A}}_n^{1/2}$ to yield OFS-adjusted samples ${\mathsf{\theta }}_{\left(j\right)}^{\mathrm{ofs}}={\widehat{\mathsf{\theta }}}_{*}+{\mathsf{\Psi }}_n({\mathsf{\theta }}_{\left(j\right)}-{\widehat{\mathsf{\theta }}}_{*})$ for all $j=1,\dots ,M$. This a-posteriori transformation is computationally efficient and simple to implement, but it does not guarantee a fully accurate characterization of the sandwich distribution.

The second method, magnitude-adjusted MCMC of Pauli et al. [31], aligns the posterior with the sandwich distribution by raising the likelihood function $L_n\left(\mathsf{\theta }\right)$ to a scalar power $0<k<1$, known as the omnibus scalar. The scalar k is chosen such that the estimated information matrices, ${\mathbf{A}}_n$ and ${\mathbf{B}}_n$ satisfy the information identity $k{\mathbf{A}}_n=k^2{\mathbf{B}}_n$. This power-likelihood approach, denoted $L_n^k\left(\mathsf{\theta }\right)$, effectively tempers the learning rate and produces posterior samples whose covariance is inversely proportional to the observed Godambe information ${\widehat{\mathcal{G}}}_n=n{\mathbf{A}}_n^{-1}{\mathbf{B}}_n^{\phantom{-1}}{\mathbf{A}}_n^{-1}$. While this method is computationally attractive, it applies a single scalar k to all d parameters. Consequently, it yields exact results only when the local shape of ${\mathcal{L}}_n\left(\mathsf{\theta }\right)$ near ${\widehat{\mathsf{\theta }}}_{*}$ is approximately quadratic. If the posterior exhibits anisotropy (directional variation) or asymmetry, a single scalar k may distort the geometry of the true sandwich distribution, suggesting the need for dimensionality-specific scaling.

The third method, curvature-adjusted MCMC of Ribatet et al. [24], modifies the sampling procedure by evaluating the likelihood $L_n\left(\mathsf{\theta }\right)$ at an affine transformation of the candidate points. Specifically, each proposed point ${\mathsf{\theta }}_{\mathrm{p}}$ is transformed to ${\mathsf{\theta }}_{\mathrm{p}}^{\mathrm{ca}}={\widehat{\mathsf{\theta }}}_{*}+{\mathbf{C}}_n({\mathsf{\theta }}_{\mathrm{p}}-{\widehat{\mathsf{\theta }}}_{*})$, where the tuning matrix is defined as ${\mathbf{C}}_n={\mathbf{B}}_n^{-1/2}{\mathbf{A}}_n^{1/2}$. This transformation effectively enforces the sandwich covariance on the MCMC samples, ensuring that the sampled chains reflect the curvature implied by the observed Godambe information. However, this method has important limitations. The matrix square roots ${\mathbf{A}}_n^{1/2}$ and ${\mathbf{B}}_n^{1/2}$ are not uniquely defined unless the log-likelihood ${\mathcal{L}}_n\left(\mathsf{\theta }\right)$ is exactly quadratic in the neighborhood of ${\widehat{\mathsf{\theta }}}_{*}$. As a result, the transformation can induce arbitrary rotations of the posterior ellipsoids, which may misrepresent the true directional asymmetries of the sandwich distribution. Moreover, curvature-adjusted MCMC does not respect parameter bounds, and care must be taken to ensure that proposed candidate points lie within the feasible domain.

As the second objective of this paper, we presented the theoretical foundation of a kernel adjustment method for sandwich-adjusted MCMC simulation. This approach is similar in spirit as magnitude-adjusted MCMC but employs a scaled log-likelihood function of the form ${\mathcal{L}}_n^{\mathrm{p}}(\mathsf{\theta }\mid \lambda )=\lambda \left(\mathsf{\theta }\right)\{{\mathcal{L}}_n\left(\mathsf{\theta }\right)-{\mathcal{L}}_n\left({\widehat{\mathsf{\theta }}}_{*}\right)\}$ centered at the maximum a posteriori (MAP) estimator ${\widehat{\mathsf{\theta }}}_{*}$ and governed by a nonconstant, parameter-dependent power $\lambda \left(\mathsf{\theta }\right)>0$. This dynamic learning rate is defined as $\lambda \left(\mathsf{\theta }\right)=\left\{{(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*})}^{\top }{\widehat{\mathcal{G}}}_n(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*})\right\}/\left\{{(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*})}^{\top }n{\mathbf{A}}_n(\mathsf{\theta }-{\widehat{\mathsf{\theta }}}_{*})\right\}$ and is typically less than one as the sandwich or Godambe information ${\widehat{\mathcal{G}}}_n$ is generally smaller in magnitude than $n{\mathbf{A}}_n$ under correct specification. Thus, the power $\lambda \left(\mathsf{\theta }\right)>0$ flattens the posterior surface in regions where the observed information exceeds the robust information, reducing overconfidence and improving robustness under model misspecification. Note that under correct specification, ${\widehat{\mathcal{G}}}_n={\widehat{\mathcal{I}}}_n=n{\mathbf{A}}_n$, we yield a unit learning rate for all $\mathsf{\theta }$ and recover the naive covariance matrix ${\mathsf{\Sigma }}_n^{\mathrm{naive}}=\widehat{\mathcal{I}}{\phantom{\underline{\widehat{\mathcal{I}}}}}_n^{-1}$. The learning rate $\lambda \left(\mathsf{\theta }\right)$ facilitates the construction of robust Bayesian credible regions under misspecification without requiring matrix square roots or eigen decompositions.

We demonstrated the four different sandwich adjustment methods by application to three case studies of increasing complexity. The first two case studies focus attention to simple statistical models with analytically tractable derivatives and offer deeper insight into the differences between naive and sandwich variance estimators under a wrong parameterization (study 1) and inadequate parametric form (study 2) of the data-generating process. The naive variance estimator fails to account for these discrepancies and leads to overconfident inference. The first study confirmed that the frequentist sandwich variance estimator produces asymptotically valid confidence intervals. The second study demonstrated that the OFS adjustment of Shaby [25] increased the spread of the naive posterior samples, but the resulting credible regions did not achieve the theoretical sandwich coverage probabilities. In contrast, parameter credible regions obtained using magnitude-, curvature-, and sandwich-adjusted MCMC simulation were in close agreement with one another and almost attained the expected coverage. This deviation was due to the assumption of symmetry in constructing the $100(1-\alpha )$% credible intervals. Altogether, the first two studies confirmed that the sandwich estimator yields asymptotically valid “robust standard errors” even when $L_n\left(\mathsf{\theta }\right)$ is wrongly parameterized or misspecified.

The third and final study applied the proposed methods to a rainfall–discharge simulation using the Xinanjiang watershed model. The results confirmed that traditional MCMC methods tend to produce overly narrow credible intervals for both model parameters and simulated outputs. This well-known phenomenon of overconditioning arises from the incorrect assumption of a well-specified model. Magnitude-, curvature-, and sandwich-adjusted MCMC simulation relax this assumption and yield substantially larger credible regions of the Xinanjiang model parameters and simulated streamflows. Our proposed method with a dynamic learning rate yields more robust Bayesian credible intervals than magnitude-adjusted MCMC sampling and does not suffer problems with nonuniqueness of principal matrix square roots as in curvature-adjusted MCMC simulation. All three methods, magnitude-, curvature-, and sandwich-adjusted MCMC methods require sample estimates ${\mathbf{A}}_n$ and ${\mathbf{B}}_{n}4$ of the sensitivity and variability matrices, respectively, along with an estimate of ${\widehat{\mathsf{\theta }}}_{*}$. In principle, the sampled chains from sandwich-adjusted MCMC converge rapidly to the sandwich distribution, since the chains are initialized in the vicinity of the MAP solution. However, this approach will always incur a greater computational cost than naive Bayesian methods.

As third and final objective of this paper, we presented an information-theoretic interpretation of the alignment score proposed by Vrugt et al. [8]. This strictly proper score measures the concordance of the bread and meat matrices and can be decomposed into a cross-entropy and differential entropy term. The misalignment score guides model improvement and enables direct comparison across models with different numbers of parameters, supporting model selection. We also explored others scalar measures for the degree of model misspecification. These include the Earth Mover’s distance, Frobenius norm, and Herfindahl index. Each measure captures different aspects of the discrepancy between naive and sandwich variance estimators caused by model misspecification. The Herfindahl index also offers a scalar diagnostic of the effective dimensionality of posterior uncertainty and serves as a useful diagnostic of anisotropy and concentration in the naive and sandwich variance estimators. Application of these measures to a suite of hydrologic models confirmed that all models were substantially misspecified. This analysis further showed that an increased model complexity does not guarantee better specification. Further research is warranted on the interplay between model complexity, the proposed misalignment score, other misspecification-based diagnostics, and widely used residual-based measures of predictive performance.