Adaptivity in Bayesian Inverse Finite Element Problems: Learning and Simultaneous Control of Discretisation and Sampling Errors

Abstract

The local size of computational grids used in partial differential equation (PDE)-based probabilistic inverse problems can have a tremendous impact on the numerical results. As a consequence, numerical model identification procedures used in structural or material engineering may yield erroneous, mesh-dependent result. In this work, we attempt to connect the field of adaptive methods for deterministic and forward probabilistic finite-element (FE) simulations and the field of FE-based Bayesian inference. In particular, our target setting is that of exact inference, whereby complex posterior distributions are to be sampled using advanced Markov Chain Monte Carlo (MCMC) algorithms. Our proposal is for the mesh refinement to be performed in a goal-oriented manner. We assume that we are interested in a finite subset of quantities of interest (QoI) such as a combination of latent uncertain parameters and/or quantities to be drawn from the posterior predictive distribution. Next, we evaluate the quality of an approximate inversion with respect to these quantities. This is done by running two chains in parallel: (i) the approximate chain and (ii) an enhanced chain whereby the approximate likelihood function is corrected using an efficient deterministic error estimate of the error introduced by the spatial discretisation of the PDE of interest. One particularly interesting feature of the proposed approach is that no user-defined tolerance is required for the quality of the QoIs, as opposed to the deterministic error estimation setting. This is because our trust in the model, and therefore a good measure for our requirement in terms of accuracy, is fully encoded in the prior. We merely need to ensure that the finite element approximation does not impact the posterior distributions of QoIs by a prohibitively large amount. We will also propose a technique to control the error introduced by the MCMC sampler, and demonstrate the validity of the combined mesh and algorithmic quality control strategy.

1. Introduction

The Bayesian statistical framework has been used extensively in the problem of system identification [1] or model updating based on experimental test data [2,3]. The objective of using inverse problems to learn or calibrate model or latent parameters, including model error terms, lies in the fact that underlying parametrised computational models are uncertain, or erroneous [4]. The experimental data when assimilated into the model, using the Bayesian inference framework, is expected to provide joint estimates of the model parameters conditional on the data and compensate for uncertainty/bias in the model predictions. However, operating only in the field of model parameters is problematic for a number of reasons. Firstly, the underlying computational models for any practical application is expensive. As a result working with a very high resolution numerical model at every stage is prohibitively expensive. Secondly, when using surrogate models in the parameter space in conjunction with a simulator, adaptive enrichment of the response surface does not take adaptive mesh refinement within its purview. This is not optimal since ensuring that the model error in the energy norm is bounded in the parameter space requires a uniformly high resolution and increases the associated computational overhead. Lastly the advanced adaptive mesh refinement techniques are rarely used in conjunction with parametric learning using Bayesian inference. There is significant room for improvement in this regard since a simultaneous control of both statistical and discretisation error would lead to substantially improved predictive numerical models both in terms of accuracy and computational efficiency.

At the root or our methodology is the estimation of errors due to the finite element approximation of the partial differential equation (PDE) of interest. Classically, spatial resolution of finite element models can be adaptively refined (also known as local h-refinement) based on a posteriori error estimation techniques [5,6,7], combined with re-meshing strategies. Within this, methods focussing on errors estimated in terms of specific quantities of interest (QoI), rather than the classical energy norm, constitute the goal-oriented adaptivity scheme [8,9,10,11,12], which is of particular interest in the present study. Numerical studies have shown that these give better convergence in the local features of the solution compared to traditional approaches.

Integrating model reduction techniques with finite element model updating techniques has received some attention in recent years [13], where the motivation is to use Bayesian model updating framework with an adaptive scheme of enriching the surrogate response surface. Multistage Bayesian inverse problems are quite important in this respect [14,15] and have important applications for system identification of vibrating systems. The prediction error is an important parameter to be calibrated in such cases, as the authors point out. But the improvement in model predictions, if solely focussed on obtaining the posterior probabilistic parameter estimates or for adaptive enrichment of the response surface in the parameter space, without considering simultaneous enhancement of the resolution of the numerical simulator would be unsatisfactory both from the standpoints of computational accuracy and efficiency.

The forward problem of uncertainty propagation has been investigated extensively for the solution of stochastically parametrised partial differential equations. These range from efficient stochastic Galerkin methods using polynomial chaos basis functions [16,17,18,19], stochastic collocation techniques [20,21], Monte-Carlo sampling based methods (and its various improvements) [22,23,24] and other deterministic sampling methods [25,26,27]. The main challenge is to obtain a good approximation of the lower order statistical moments of the state vector or specific quantities of interest. On the other hand, advances in the resolution of stochastic inverse problems has become a very active topic in engineering and mathematical research (see e.g., [28]). Scalable Bayesian inversion algorithms for large-scale problems have been investigated [29] and as well as Bayesian inversion to probabilistic robust optimization under uncertainty [30]. The use of adaptive sparse-grid surrogates unified with Bayesian inversion for posterior density estimates of the model or design has also been studied parameters [3,31,32]. This research focuses mostly on the definition of surrogates of the numerical model and having adaptive methods to control the statistical sampling error in the definition of the response surface. Lately there has been some research [33] that defines adaptivity as the local enrichment of the surrogate model and uses error estimation to bound this error, with no mention of spatial discretisation. However, coupling engineering uncertainty quantification (UQ) with an adaptive scheme for goal-oriented finite element model refinement remains a sparsely studied domain and presents significant challenges. It is so both from the problem formulation perspective, owing to the choice of appropriate candidate estimates based on which adaptive model enrichment can be performed, as well as incorporating it into the general formulation of Bayesian inversion.

The main focus of the paper is to develop a robust methodology for the simultaneous control of errors from multiple sources—the goal-oriented finite element error and the uncertainty-driven statistical error—in a Bayesian identification framework for identification system parameters conditional on data (experimental or otherwise). The novel algorithmic approach proposed in this paper consists in running two Markov Chain Monte Carlo algorithms simultaneously to sample the posterior densities of the quantities of interest (component-wise MCMC). The first chain utilises the current finite element model, whilst the second chain runs with a corrected likelihood function that takes into account the discretisation error. The latter quantity may be constructed by making use of a posteriori finite element error estimates available in the literature. At any time during the sampling process, two empirical densities are available and may be compared to evaluate the effect of the discretisation error onto posterior densities. The second building block of our algorithmic methodology allows us to determine whether enough samples have been drawn by the component-wise MCMC. Using multiple parallel chains combined with bootstrap-based estimates of sampling errors, we automatically stop the MCMC algorithms when either (i) the required level of accuracy is achieved or (ii) we have generated enough statistical confidence in the fact that the discretisation error is too large for our purpose, and therefore that mesh refinement is necessary. While any available error estimate of the discretisation error may be used within the general strategy outlined above (a goal-oriented residual or recovery-based error estimate, for instance [8,9,11,34]), we choose instead to construct this estimate via a dedicated machine learning approach. This interesting feature, inspired by previous work in data-driven error modelling [35,36,37,38], will be briefly outlined in the paper.

The paper is organised as follows. Section 2 introduces the Bayesian inverse problem based on a finite element model of a parametrised vibrating structure. This section also includes discussions on discretisation error. Some numerical examples are presented in Section 3, which aims to demonstrate the convergence of the joint posterior distributions on the model parameters through successive stages of mesh refinement. Section 4 discusses the total error as a combination of statistical and discretisation error that results from the MCMC algorithm used for sampling from posterior distributions. Section 5 gives the methodology for robust, simultaneous control of all error sources within the adaptive inverse problem solver using a component-wise MCMC algorithm in conjunction with bootstrap-aggregated regression model for model parameters. Numerical examples are presented and discussed in Section 6 to demonstrate the capabilities of the proposed methodology.

2. Finite Element Bayesian Inverse Problems

Although the methodology proposed in this paper is general, we will apply it to the verification of popular inverse finite element procedures used to monitor the integrity of structures during service. We will assume that the structure can be modelled as an elastic body, and that potential structural damage can be modelled by the evolution of a field of elastic constants characterised by a finite number of uncertain parameters. Assuming that the resonance frequencies of the structure have been measured physically, we attempt to identify this set of mathematical parameters, through the solution of an inverse problem. Significant deviations from the parameter values corresponding to an undamaged structural state may reveal structural failure. This is a typical condition monitoring method used for non-destructive structural testing.

In order to clarify the proposed study, we will assume that we have measured the $n_{\mathrm{d}}$ first dynamic eigenfrequencies $d={\left(\begin{array}{ccc}ln\left({\omega }_1\right)& ...& ln\left({\omega }_{n_{\mathrm{d}}}\right)\end{array}\right)}^T$ of the structure represented in Figure 1. The purpose of the model inversion is to identify the values of $n_{\mu }$ parameters $\mu ={\left(\begin{array}{ccc}{\mu }_1& ...& {\mu }_{n_{\mu }}\end{array}\right)}^T$ of the structural dynamics model, here the log of the elastic modulii of the subdomains represented in Figure 1. We will also aim to predict the remaining $n_{\mathrm{p}}-n_{\mathrm{d}}$ frequencies $p={\left(\begin{array}{ccc}ln\left({\omega }_{n_{\mathrm{d}}+1}\right)& ...& ln\left({\omega }_{n_{\mathrm{d}}+n_{\mathrm{p}}}\right)\end{array}\right)}^T$. The relationships $\mu \to d\left(\mu \right)$ and $\mu \to p\left(\mu \right)$, i.e., the computational model, are defined implicitly through the evaluation of a standard finite element model of the steady-state, undamped structural vibrations.

2.1. Bayesian Inverse Problem

We assume that the quantities $d$ that are measured experimentally are described by the mathematical model, up to an error, which is modelled in a probabilistic manner. Following standard Bayesian procedures, this error is modelled as a white Gaussian noise,

$$d=d_{\mathrm{m}}+ϵ,ϵ\sim \mathcal{N}(0,{\Sigma }_{\mathrm{n}}^{-1})$$

(1)

with ${\Sigma }_0$ a positive definite covariance matrix. $A_{\mathrm{d}}:\mu \to d_{\mathrm{m}}:=A_{\mathrm{d}}\left(\mu \right)$ is the mathematical prediction that corresponds to the experimental observation. Model $A_{\mathrm{d}}$ is a deterministic function of $n_{\mu }$ uncertain model parameters that we organise in a $n_{\mu }$-dimensional vector $\mu$. Bayesian inversion requires to associate a prior probability density with the model parameters $\mu$. This prior probability density, denoted by ${\pi }_{\mu }\left(\mu \right)$ in the following, encodes the knowledge that we possess about $\mu$ before making any physical observations.

Seen from a different point of view, the probabilistic inversion setting amounts to the definition of a joint probability distribution for $\mu$ and $d$:

$${\pi }_{\mu ,d}(\mu ,d)={\pi }_{d|\mu }(\mu ,d){\pi }_{\mu }\left(\mu \right)$$

(2)

The formal expression of likelihood function $\mathcal{L}(\mu ;d):={\pi }_{d|\mu }(\mu ,d)$ is a direct consequence of assumption (1), namely

$$\mathcal{L}(\mu ;d)=\frac{1}{{\displaystyle \sqrt{{\left(2\pi \right)}^{n_{\mathrm{d}}}\left|{\Sigma }_{\mathrm{n}}\right|}}}{e}^{{\displaystyle -\frac{1}{2}\left({\left(d-A_{\mathrm{d}}\left(\mu \right)\right)}^T{\Sigma }_{\mathrm{n}}^{-1}\left(d-A_{\mathrm{d}}\left(\mu \right)\right)\right)}}$$

(3)

It is now possible to formally proceed to the inversion itself by conditioning the joint distribution to actually observed quantities. Applying Bayes’ formula, the posterior probability density of the model parameters is

$${\pi }_{\mu |d}(\mu ;d)=\frac{\mathcal{L}(\mu ;d){\pi }_{\mu }\left(\mu \right)}{{\pi }_d\left(d\right)}$$

(4)

where ${\pi }_d\left(d\right)$ is a normalising constant, whose computation requires a usually intractable integration. Bayes formula provides us with an updated knowledge about the uncertain part of our mathematical model. It is now possible to predict unobserved quantities i.e.,

$$p=A_{\mathrm{p}}\left(\mu \right),\mu \sim {\pi }_{\mu }|d(\mu ;d)$$

(5)

by formally propagating the posterior uncertainty through model $A_{\mathrm{p}}:\mu \to p_{\mathrm{m}}:=A_{\mathrm{p}}\left(\mu \right)$. The posterior predictive probability density of $p$ will be denoted by symbol ${\pi }_{p|d}(\mu ;d)$.

2.2. Finite Element Modelling of Inverse Structural Vibration Problems

2.2.1. Direct Finite element Procedure for Frequency-Domain Vibrations

The numerical model

$$\mu \to A\left(\mu \right):=\left(\begin{array}{c}{A}_{\mathrm{d}}\left(\mu \right)\\ A_{\mathrm{p}}\left(\mu \right)\end{array}\right)$$

(6)

is implicitly defined through the solution of a continuum mechanics problem. The components of $d_{\mathrm{m}}$ and $p$ are the logarithm of the eigenvalues corresponding to the following parametrised eigenvalue problem: Find $u\in {\mathcal{H}}^1(\Omega )$ and $\omega \in {\mathbb{R}}^{+}$ such that $\forall v\in {\mathcal{H}}^1(\Omega )$

$${\int }_{\Omega }{\nabla }_{\mathrm{s}}u:\mathbb{D}(x;\mu ):{\nabla }_{\mathrm{s}}vd\Omega -{\omega }^2{\int }_{\Omega }\rho u·vd\Omega =0$$

(7)

In the previous variational statement, $\Omega$ is the domain occupied by the structure of interest, ${\mathcal{H}}^1(\Omega )$ is the space of functions defined over $\Omega$, with values in ${\mathbb{R}}^2$, that are zero on part ${\partial }_{\mathrm{u}}\Omega$ of the boundary of the domain, and whose derivatives up to order one are square integrable. ${\nabla }_{\mathrm{s}}$ denotes the symmetric part of the gradient operator. $\mathbb{D}$ is the fourth order Hooke tensor and $\rho$ is the mass density of the solid material. The problem possesses an infinite number of solutions $(\omega ,u)$ called free vibration modes. We order the free vibration frequencies in increasing order ${\omega }_1<{\omega }_2<...<{\omega }_{n_{\mathrm{d}}+n_{\mathrm{p}}}$. The free vibration modes are functions of uncertain material parameters $\mu$ through the definition of the Hooke tensor. Specifically,

$$\mathbb{D}(x;\mu ):{\nabla }_{\mathrm{s}}.=\lambda (x;\mu )\mathrm{Tr}({\nabla }_{\mathrm{s}}.)\mathbb{I}+2G(x;\mu )({\nabla }_{\mathrm{s}}.)$$

(8)

with the parametrised Lamé constants

$$\lambda (x;\mu )=\frac{0.3\times E(x;\mu )}{2(1+0.3)(1-2\times 0.3)}\mathrm{and}G(x;\mu )=\frac{E(x;\mu )}{2(1+0.3)}$$

(9)

Solving the continuum mechanics model is equivalent to evaluating the mapping

$$\mu \to \left(\begin{array}{c}{A}_{\mathrm{d}}\left(\mu \right)\\ A_{\mathrm{p}}\left(\mu \right)\end{array}\right)=\left(\begin{array}{c}ln{\left(\begin{array}{ccc}{\omega }_1\left(\mu \right)& ...& {\omega }_{n_{\mathrm{d}}}\left(\mu \right)\end{array}\right)}^T\\ ln{\left(\begin{array}{ccc}{\omega }_{n_{\mathrm{d}}+1}\left(\mu \right)& ...& {\omega }_{n_{\mathrm{d}}+n_{\mathrm{p}}}\left(\mu \right)\end{array}\right)}^T\end{array}\right).$$

(10)

There is, in general, no analytical solution to the continuous vibration problem, and a standard way to obtain approximate solutions is to substitute a finite element space ${\mathcal{U}}^h(\Omega )$ for infinite dimensional search space ${\mathcal{H}}^1(\Omega )\supset {\mathcal{U}}^h(\Omega )$ [39]. Too coarse a finite element discretisation may result in poorly predictive results, while too fine a mesh will lead to numerically intractable results, or, in any case, to a waste of computing resources.

2.2.2. Finite Element Approximation of the Bayesian Inverse Problem

Whilst using the continuum mechanics model exactly would deliver the posterior density ${\pi }_{\mu |d}(\mu ;d)$ as solution of the Bayesian inverse problem, we now have an approximate posterior density

$${\overline{\pi }}_{\mu }|d(\mu ;d)=\frac{\overline{\mathcal{L}}(\mu ;d){\pi }_{\mu }\left(\mu \right)}{\overline{\pi }\left(d\right)}$$

(11)

where the finite element likelihood $\overline{\mathcal{L}}$ is obtained by substituting finite element mapping ${\overline{A}}_{\mathrm{d}}$ for $A_{\mathrm{d}}$ in Equation (3). Similarly, the approximate posterior density of $p$ is denoted by ${\overline{\pi }}_{p|d}(\mu ;d)$ and obtained by evaluating finite element mapping ${\overline{A}}_{\mathrm{p}}$ instead of $A_{\mathrm{p}}$ when propagating the posterior uncertainties forward.

2.2.3. Discretisation Error

The finite element error is the mismatch between ${\overline{\pi }}_{\mu |d}(\mu ;d)$ and ${\pi }_{\mu |d}(\mu ;d)$ on the one hand, and ${\overline{\pi }}_{p|d}(\mu ;d)$ and ${\pi }_{p|d}(\mu ;d)$ on the other hand. Various measures can be used to quantify this mismatch, amongst which the Hellinger distance, defined by

$${\mathcal{D}}_{\mathrm{h}}({\pi }_{\mu |d}\left(\mu \right),{\overline{\pi }}_{\mu |d}\left(\mu \right))={∥\sqrt{{\pi }_{\mu |d}\left(\mu \right)}-\sqrt{{\overline{\pi }}_{\mu |d}\left(\mu \right)}∥}_2$$

(12)

the Kullback-Leibler divergence, the total variation distance and the Kolmogorov-Smirnov (KS) distance, defined by

$${\mathcal{D}}_{\mathrm{ks}}({\pi }_{\mu |d}\left(\mu \right),{\overline{\pi }}_{\mu |d}\left(\mu \right))=\underset{\mu }{sup}\left(\left|({\Pi }_{\mu |d}\left(\mu \right)-{\overline{\Pi }}_{\mu |d}\left(\mu \right))\right|\right)$$

(13)

where the capital symbols ${\Pi }_{.}$ denotes the cumulative probability density corresponding to ${\pi }_{.}$, and ${\overline{\Pi }}_{.}$ is the finite element approximation of $\Pi$. This contribution will make use of the latter measure, in a one-dimensional setting (i.e., it will be applied to control the accuracy of the posterior density of one of the elements of $\mu$ or one of the elements of $p$). The attractiveness of the Kolmogorov–Smirnov distance is its straightforward application in the context of Monte-Carlo procedures, where only empirical densities are available, and its closeness to confidence intervals (CIs), which makes its values relatively easy to interpret within the context of a posteriori error estimation. Notice that the ${\mathcal{D}}_{\mathrm{ks}}$ is lower bounded by 0 (identical density functions) and upper bounded by 1 (non-overlapping support for the probability density functions).

3. Numerical Examples—Part I: Effect of Discretisation Errors Onto Posterior Densities

This section introduces the numerical examples that will be investigated in this paper, and aims to provide a first qualitative understanding of the effect of mesh refinement onto the quality of posterior probability densities.

3.1. Forward Stochastic Model

The stochastic field of Young’s modulus that will be used to exemplify the error control approach proposed in this paper is defined via a decomposition of domain $\Omega$ into $n_{\mu }+1=7$ non-overlapping subdomains ${\displaystyle {\left\{{\Omega }_i\right\}}_{〚1n_{\mu }+1〛}}$ such that ${\displaystyle \bigcup _{i\in 〚1n_{\mu }+1〛}{\Omega }_i=\Omega }$ and ${\displaystyle {\Omega }_i\cap {\Omega }_j=\left\{\right\}}$ for $i\ne j$. The domain decomposition is represented in Figure 1. Then, the proposed model is such that

$$E(x;\mu )=e^{{\mu }_i}\mathrm{if}x\in {\Omega }_i\mathrm{and}i\ne 1,E(x;\mu )=1\mathrm{otherwise}.$$

(14)

Hence, the scalar parameters contained in vector $\mu \in {\mathbb{R}}^{n_{\mu }}$ are the logarithms of the Young’s modulii corresponding to each of the subdomains.

The prior density is a multivariate Gaussian, and is given by equation

$${\pi }_{\mu }\left(\mu \right)=\frac{1}{{\displaystyle \sqrt{{\left(2\pi \right)}^{n_{\mu }}\left|{\Sigma }_0\right|}}}{e}^{{\displaystyle -\frac{1}{2}\left({\left(\mu -{\mu }_0\right)}^T{\Sigma }_0^{-1}\left(\mu -{\mu }_0\right)\right)}}$$

(15)

The prior mean ${\mu }_0$ is the null vector, and the prior variance ${\Sigma }_0={\sigma }_0^2{I}_d$, where $I_d\in {\mathbb{R}}^{n_{\mu }}\times {\mathbb{R}}^{n_{\mu }}$ is the identity matrix, is diagonal and isotropic. The prior density is represented in dimensions $({\mu }_1,{\mu }_3)$ in Figure 1.

Finally, we model the error $ϵ$ as a zero-mean multivariate Gaussian (consistently with what was described in previous section), with independent components and isotropic variance, i.e., ${\Sigma }_{\mathrm{n}}={\sigma }_n^2{I}_d$.

3.2. Computational Meshes

The evaluation of the likelihood function appearing in solution (11) of the Bayesian inverse problem requires solving the continuum mechanics problem using the finite element method. In this example, we use a sequence of meshes ${\left\{{\mathcal{M}}_i\right\}}_{i\in 〚n_m〛}$ associated with a monotonically increasing number of degrees of freedom. These meshes are represented in Figure 2. Although the sequence of meshes is not strictly hierarchical, we see that the typically uniform element size is divided by $\approx 2$ when moving from mesh ${\mathcal{M}}_i$ to mesh ${\mathcal{M}}_{i+1}$. The intermediate meshes ${\left\{{\mathcal{M}}_{i+\frac{1}{2}}\right\}}_{i\in 〚1{\tilde{n}}_m〛}$ represented in Figure 2 will be used later on.

3.3. Inverse Problems and First Results

Two tests will now be investigated:

• Test 1(weakly informative data): only the first eigenvalue is measured, i.e., $d$ is scalar. This can be interpreted as a task of model updating, where new data is used to update an existing knowledge.
• Test 2 (strongly informative data): the first three eigenvalues of the structure are measured. This can be interpreted as an inverse problem, where rich information is used to identify all the unknown of the model, and the probabilistic setting acts as a regulariser.

The two corresponding marginal posterior density ${\widehat{\pi }}_{{\mu }_1,{\mu }_3|d}$ obtained when using mesh ${\mathcal{M}}_3$ are represented in Figure 3. Samples from this density are obtained by a Monte-Carlo sampler, as presented in next section. A Kernel Density Estimate (KDE) is used as a smoother for illustration purposes only. Notice that for the predictive posterior densities, the histograms correspond to the marginal densities of each individual eigenvalue, which explains their overlap.

It is interesting to notice that the posterior densities observed in Test 2 are much sharper than that of Test 1, owing to the quality of the data. The symmetry in the results are a consequence of structural and probabilistic symmetries (see Figure 1 and the definition of the prior probability density).

Synthetic data for this problem is generated by computing the average of the spectra delivered by meshes ${\mathcal{M}}_2$ and ${\mathcal{M}}_{2+\frac{1}{2}}$, for a reference parameter vector ${\mu }_{\mathrm{d}}\ne {\mu }_0$ but situated in the vicinity of the prior mode. The model averaging, together with selecting relatively coarse finite element meshes in sequence ${\left\{{\mathcal{M}}_i\right\}}_{i\in 〚n_m〛}$, is meant to circumvent the “inverse crime" problem.

3.4. Convergence with Mesh Refinement

The posterior densities corresponding to Test 2 and to various level of mesh refinement are displayed in Figure 4. The difference between the modes of the predicted eigenvalues that are obtained with coarse and fine meshes are very large. The fact that this discrepancy increases with the mode number is to be expected as the spatial wave length of the deformations in the continuum can be shown to increase linearly with the eigenfrequency. As a consequence, a good mesh for the first range of frequencies might be unable to capture the faster spatial variations associated with higher vibration mode.

The difference between the posteriori densities corresponding to meshes ${\mathcal{M}}_3$ and ${\mathcal{M}}_4$ is qualitatively small. The solution of the Bayesian inverse problem converges with mesh refinement. Notice that the posterior distribution for the model parameters goes from mono-modal to multi-modal, which may prove a stumbling block when selecting an appropriate Monte-Carlo sampler.

4. Monte-Carlo Sampler and Combined Effect of Statistical and Discretisation Errors

4.1. Tempered Metropolis-Hastings Markov Chain Monte Carlo Algorithm

The approximate posterior density described by Equation (11) is an arbitrarily complex function of $\mu$, as it depends on the nonlinear computational model ${\left(A_{\mathrm{d}}{\left(\mu \right)}^T{A}_{\mathrm{p}}{\left(\mu \right)}^T\right)}^T$. In particular, standard random number generators cannot be used to draw samples from this distribution. Importance Monte-Carlo procedures whereby the prior distribution is used as proposal density, will also fail. This is because the posterior density may be arbitrarily different from the prior density, resulting in unacceptably large variances of importance sampling estimates. Designing better proposal densities a priori is impossible and a successful importance sampling approach would require generating the proposal density using advanced methods such as sequential Monte-Carlo samplers.

One of the simplest and generic generator of samples from posterior densities is the Metropolis-Hastings (MH) Markov Chain Monte Carlo (MCMC) algorithm (see Figure 5). The algorithm works as follows. Starting from sample ${\mu }_n$, the next sample ${\mu }_{n+1}$ is drawn from transitional distribution

$$T_{{\mu }_{n+1}\leftarrow {\mu }_n}({\mu }_{n+1};{\mu }_n)\propto min\left(1,\frac{{\overline{\pi }}_{\mu |d}\left({\mu }_{n+1}\right)}{{\overline{\pi }}_{\mu |d}\left({\mu }_n\right)}\right)\mathcal{N}({\mu }_{n+1};{\mu }_n,\tilde{\Sigma })$$

(16)

This is done by first drawing a random move from ${\mu }_k$ using the multivariate Gaussian (any proposal can be used, but the expressions exposed in this section are only valid for symmetric proposals), and then accepting or rejecting the move in order to account for the first term of the transition. Typically, a move ending up in a state of higher posterior density is always accepted, whilst a move ending up in a state of lower density may be accepted or not, depending on the result of a die and the ratio of posterior densities between current and proposed states.

This particular transition is designed such that the following ergodic property holds:

$${\int }_{{\mu }_{n+1}}{T}_{{\mu }_{n+1}\leftarrow {\mu }_n}({\mu }_{n+1};{\mu }_n)d{\mu }_n={\overline{\pi }}_{\mu |d}\left({\mu }_{n+1}\right)$$

(17)

As a result, under some assumptions, the Markov chain is guaranteed to have ${\overline{\pi }}_{\mu |d}$ as its stationary distribution, which means that ${\mu }_n\sim {\overline{\pi }}_{\mu |d}$ as $n\to \infty$. Each sample, taken individually, is distributed according to the posterior density, if the chain is run long enough. Due to the Markov process, the samples are not independent, which makes frequentist error estimation and convergence diagnosis more difficult than in the context of traditional Monte-Carlo algorithm, where tools such as standard error and bootstrap apply without particular difficulty. Practically, a burn-in phase is first observed, whereby the chain “seeks” the regions of high probability densities (see Figure 5). Once found, samples become progressively distributed according to the target posterior density.

Notice that $\tilde{\Sigma }$ is difficult to choose a priori. Adaptive proposal MCMC have been proposed in [40,41]. Alternative solutions to choose good proposal are methods based on particle mechanics (e.g., Langevin diffusion, Hamiltonian dynamics) (see e.g., [42]). In this particular contribution, the proposal densities have been calibrated “by hand”. We use a tempered version of the MCMC algorithm, whereby we simultaneously sample multiple tempered replicates of the posterior density, with proposed state exchange between replicates that are subsequently Metropolis corrected [43,44] (see also [45] for an interesting application in the context of structural damage assessment). This relatively classical MCMC algorithm allows us to sample from; densities that are multi-modal or become multi-modal with mesh refinement (at least in the low-dimensional parametric settings that is investigated in this paper, as multi-modality in high-dimensions remains an open problem).

The call to the finite element solver is hidden in the acceptance/rejection test, which requires the evaluation of the finite element likelihood at the proposed state. Therefore, as for a standard MC algorithm, one iteration means one evaluation of the computational model.

4.2. Empirical Posterior Densities

Once n samples $\tilde{\mathcal{M}}=\{{\mu }_1,{\mu }_2,...,{\mu }_n\}$ have been computed (n larger than 10) by the MH algorithm, we discard the first 25% (burn-in) of these samples, resulting in $\tilde{n}$ samples that we hope are located in the ergodic part of the Markov chain. The resulting empirical probability density is given by

$${\overline{\pi }}_{\mu |d}\left(\mu \right)\approx {\overline{\pi }}_{\mu |d}^{\left(\mathrm{n}\right)}\left(\mu \right):=\frac{1}{\tilde{n}}\sum _{k=n-\tilde{n}+1}^n\delta (\mu -{\mu }_k)$$

(18)

where $\delta$ is the Dirac delta function. Accordingly, the empirical predictive posterior density is

$${\overline{\pi }}_{p|d}\left(p\right)\approx {\overline{\pi }}_{p|d}^{\left(\mathrm{n}\right)}\left(p\right):=\frac{1}{\tilde{n}}\sum _{k=n-\tilde{n}+1}^n\delta (p-A_{\mathrm{p}}\left({\mu }_k\right))$$

(19)

The cumulative empirical predictive posterior density may be expressed as

$${\overline{\Pi }}_{\mu |d=d}\left(\mu \right):=\frac{1}{\tilde{n}}\sum _{k=n-\tilde{n}+1}^n\mathbb{I}(\mu ;{\mu }_k\le \mu )$$

(20)

where $\mathbb{I}$ is the indicator function, and the inequality is to be understood in a component-by-component manner. A similar definition holds for the cumulative posterior density of $p$.

4.3. Total Error Measure

It is now clear that both the finite element discretisation and the Monte-Carlo approximate sampling affect the quality of the resulting posterior densities. The total error, for the marginal distribution of a single element ${\mu }_i$ of $\mu$, reads as

$${\mathcal{D}}_{\mathrm{ks}}({\pi }_{{\mu }_i|d}\left({\mu }_i\right),{\overline{\pi }}_{{\mu }_i|d}^{\left(\mathrm{n}\right)}\left({\mu }_i\right))=\underset{{\mu }_i}{sup}\left(\left|({\Pi }_{{\mu }_i|d}\left({\mu }_i\right)-{\overline{\Pi }}_{{\mu }_i|d}^{\left(\mathrm{n}\right)}\left({\mu }_i\right))\right|\right)$$

(21)

which can be formally decomposed as follows

$$\begin{array}{ccc}\hfill {\displaystyle {\mathcal{D}}_{\mathrm{ks}}({\pi }_{{\mu }_i|d}\left({\mu }_i\right),{\overline{\pi }}_{{\mu }_i|d}^{\left(\mathrm{n}\right)}\left({\mu }_i\right))}& =& {\displaystyle \underset{{\mu }_i}{sup}\left(\left|({\Pi }_{{\mu }_i|d}\left({\mu }_i\right)-{\overline{\Pi }}_{{\mu }_i|d}\left({\mu }_i\right))+{\overline{\Pi }}_{{\mu }_i|d}\left({\mu }_i\right)-{\overline{\Pi }}_{{\mu }_i|d}^{\left(\mathrm{n}\right)}\left({\mu }_i\right))\right|\right)}\hfill \\ & \le & {\displaystyle {\mathcal{D}}_{\mathrm{ks}}({\pi }_{{\mu }_i|d}\left({\mu }_i\right),{\overline{\pi }}_{{\mu }_i|d}\left({\mu }_i\right))+{\mathcal{D}}_{\mathrm{ks}}({\overline{\pi }}_{{\mu }_i|d}\left({\mu }_i\right),{\overline{\pi }}_{{\mu }_i|d}^{\mathrm{mc}}\left({\mu }_i\right))}\hfill \end{array}$$

(22)

In the last expression, the first term is the pure finite element error, which would occur if we could run the Markov process for an infinite number of iterations, while the second term is a pure statistical error.

We also define the error of an element $p_i$ of $p$ as follows:

$${\mathcal{D}}_{\mathrm{ks}}({\pi }_{p_i|d}\left(p_i\right),{\overline{\pi }}_{p_i|d}^{\left(\mathrm{n}\right)}\left({\mu }_i\right))=\underset{p_i}{sup}\left(\left|({\Pi }_{p_i|d}\left(p_i\right)-{\overline{\Pi }}_{p_i|d}^{\left(\mathrm{n}\right)}\left(p_i\right))\right|\right)$$

(23)

5. Robust, Automatised and Comprehensive Error Control

5.1. Simulation of the Discretisation Error

The finite element method introduces an error in computational mapping $A\left(\mu \right)={\left(\begin{array}{cc}{A}_{\mathrm{d}}{\left(\mu \right)}^T& A_{\mathrm{p}}{\left(\mu \right)}^T\end{array}\right)}^T$. Mapping $A$ contains all the scalar quantities that need to be evaluated through calls to the finite element solver, namely the numerical predictions of the physical measurements $d$, and the posterior predictions $p$. We define the error in the simulated data as

$$\Delta A_{\mathrm{d}}\left(\mu \right)=A_{\mathrm{d}}\left(\mu \right)-{\overline{A}}_{\mathrm{d}}\left(\mu \right)$$

(24)

and the error in the posteriori predictions as

$$\Delta A_{\mathrm{p}}\left(\mu \right)=A_{\mathrm{p}}\left(\mu \right)-{\overline{A}}_{\mathrm{p}}\left(\mu \right)$$

(25)

For now, we assume that both these quantities can be estimated, at affordable numerical cost and in a reliable manner. Therefore, for any value of parameter $\mu$, a corrected computational model is available, which reads as

$$\left(\begin{array}{c}{A}_{\mathrm{d}}\left(\mu \right)\\ A_{\mathrm{p}}\left(\mu \right)\end{array}\right)\underset{\mathrm{very}\mathrm{close}\mathrm{to}}{\approx }\left(\begin{array}{c}{\widehat{A}}_{\mathrm{d}}\left(\mu \right)\\ {\widehat{A}}_{\mathrm{p}}\left(\mu \right)\end{array}\right):=\left(\begin{array}{c}{\overline{A}}_{\mathrm{d}}\left(\mu \right)\\ {\overline{A}}_{\mathrm{p}}\left(\mu \right)\end{array}\right)+\left(\begin{array}{c}{\widehat{\Delta A}}_{\mathrm{d}}\left(\mu \right)\\ {\widehat{\Delta A}}_{\mathrm{p}}\left(\mu \right)\end{array}\right)$$

(26)

where symbol $\widehat{.}$ denotes computable estimates.

5.2. Component-Wise MCMC

It is now posible to sample the corrected posterior distribution

$${\widehat{\pi }}_{\mu |d}(\mu ;d)=\frac{\widehat{\mathcal{L}}(\mu ;d){\pi }_{\mu }\left(\mu \right)}{\widehat{\pi }\left(d\right)}$$

(27)

using MCMC. It should be clear that the corrected posterior distribution is simply obtained by replacing the corrected computational model (26) into the expression of the likelihood function, Equation (3). Notice that the normalising constant is affected by modifications of the computational model. This is of no practical consequence as MCMC samplers work with unnormalised densities, and the KS distance uses empirical cumulative distributions directly, without the need for smoothing or marginalisation.

We formally define a component-wise MCMC were the uncorrected and corrected computational models are sampled at the same time. This will yield an estimate of the effect of the discretisation error on posteriori densities at any stage of the Markov process, which, in turn, will allow us to develop an early-stopping methodology. The Component-wise MCMC iteration proceeds as follows, given a current sample $({\overline{\mu }}_n,{\widehat{\mu }}_n)$ of the uncorrected/corrected finite element posteriori densities,

• Draw $({\overline{\mu }}_{n+1},{\widehat{\mu }}_{n+1})$ such that

  $${\overline{\mu }}_{n+1}\sim \mathcal{N}(.;{\overline{\mu }}_n,\tilde{\Sigma })\mathrm{and}{\widehat{\mu }}_{n+1}\sim \mathcal{N}(.;{\widehat{\mu }}_n,\tilde{\Sigma })$$

  (28)
• draw $(u,v)$ such that

  $$u\sim \mathcal{U}\left(\right[01\left]\right)\mathrm{and}v\sim \mathcal{U}\left(\right[01\left]\right)$$

  (29)
• Accept ${\overline{\mu }}_{n+1}$ if and only if

  $$u\le min\left(1,\frac{\overline{\mathcal{L}}({\overline{\mu }}_{n+1};d){\pi }_{\mu }\left({\overline{\mu }}_{n+1}\right)}{\overline{\mathcal{L}}({\overline{\mu }}_n;d){\pi }_{\mu }\left({\overline{\mu }}_n\right)}\right)$$

  (30)

  set ${\overline{\mu }}_{n+1}={\overline{\mu }}_k$ otherwise.
• Accept ${\widehat{\mu }}_{n+1}$ if and only if

  $$v\le min\left(1,\frac{\widehat{\mathcal{L}}({\widehat{\mu }}_{n+1};d){\pi }_{\mu }\left({\widehat{\mu }}_{n+1}\right)}{\widehat{\mathcal{L}}({\widehat{\mu }}_n;d){\pi }_{\mu }\left({\widehat{\mu }}_n\right)}\right)$$

  (31)

  set ${\widehat{\mu }}_{n+1}={\widehat{\mu }}_k$ otherwise.

The MCMC algorithm is initialised by state $({\overline{\mu }}_0,{\widehat{\mu }}_0)$, where both $({\overline{\mu }}_0$ and ${\widehat{\mu }}_0)$ are drawn from distribution ${\pi }_0$.

In the field of a posteriori finite element error estimation, the error estimate is usually a post-processing operation of the coarse finite element solution. This adapts seamlessly to non-Markovian Monte-Carlo samplers, by post-processing each of the independent samples. Here, unfortunately, the finite element model has to be called twice at every iteration of the Markov process. Whether this can be avoided or not, for instance by making use of elements of sequential Monte-Carlo samplers, is unclear to us at this stage.

5.3. Machine Learning-Based Simulation of the Discretisation Error

At this stage, any numerical error estimator can be used, provided that it is goal-oriented. A method of choice could be a residual-based [5,6] or smoothing-based a posteriori error estimator [12,46] in conjunction with the adjoint methodology [8,9,11]. In addition, nothing prevents us from using a meta-modelling approach, such as projection-based reduced order modelling [47,48,49,50,51,52] or polynomial chaos expansions [16,17] to approximate the variations of the computed quantities of interest with parameter variation. Error estimates also exist for such two-level approximations.

In this contribution, we develop and use a feature-based method, that finds its roots in data-science methodologies and is much more “black-box” than the previously mentioned strategies. The proposed technique is inspired by the work of [35,36,37].

The exact continuum mechanics model is well approximated by a very refined numerical strategy (e.g., no meta-modelling and a very fine mesh). However, this very refined numerical model cannot be used at every iteration of MCMC as its evaluation is very costly. We propose to train a model that will map parameter $\mu$ to the output of the generally intractable very fine model through the combination of (i) a dedicated feature extractor and (ii) a weakly parametric regression model. This combination is defined as

$$\left(\begin{array}{c}{A}_{\mathrm{d}}^{★}\left(\mu \right)\\ A_{\mathrm{p}}^{★}\left(\mu \right)\end{array}\right)-\left(\begin{array}{c}{\overline{A}}_{\mathrm{d}}\left(\mu \right)\\ {\overline{A}}_{\mathrm{p}}\left(\mu \right)\end{array}\right)\underset{\mathrm{wished}}{=}\left(\begin{array}{c}{\widehat{\Delta A}}_{\mathrm{d}}\left(\mu \right)\\ {\widehat{\Delta A}}_{\mathrm{p}}\left(\mu \right)\end{array}\right)\underset{\mathrm{constructed}}{=}\mathcal{R}(\mathcal{F}\left(\mu \right);\theta )$$

(32)

where the ${.}^{★}$ denote quantities that are delivered by the overkill (but computable) numerical model and $\mathcal{R}$ is a weakly parametrised regression model, here a neural network regression (a Gaussian process could be used as well, but a random forest would probably have been the most efficient choice, given the way we bootstrap the regression model to generate estimates of generalisation errors), parametrised by a set of parameters $\theta \in {\mathbb{R}}^{n_{\theta }}$ (we do not explicitly distinguish parameters and hyper-parameters in our notations). $\mathcal{F}$ is a mapping from input $\mu$ to a feature space. Its careful design is critical to the success of the machine learning procedure. We choose to construct the following features:

$$\mathcal{F}\left(\mu \right)=\left(\begin{array}{c}{\displaystyle {\tilde{A}}_{\mathrm{d}}\left(\mu \right)-{\overline{A}}_{\mathrm{d}}\left(\mu \right)}\\ {\displaystyle {\overline{A}}_{\mathrm{d}}\left(\mu \right)}\\ {\displaystyle {\tilde{A}}_{\mathrm{p}}\left(\mu \right)-{\overline{A}}_{\mathrm{p}}\left(\mu \right)}\\ {\displaystyle {\overline{A}}_{\mathrm{p}}\left(\mu \right)}\end{array}\right)$$

(33)

where ${\tilde{A}}_{\mathrm{d}}\left(\mu \right)$ are slighlty corrected computational models, here generated by refining the mesh by a moderate factor. In our examples, the typical mesh size is divided by 1.5 (see Figure 2 and Figure 6 where we use mesh ${\mathcal{M}}_{i+\frac{1}{2}}$ to correct mesh ${\mathcal{M}}_i$, whilst the overkill solution is computed using ${\mathcal{M}}_{i+2}$). In this fashion, the generation of features will remain of the order of the computation of the finite element solution itself.

Now, we train $n_{\mathrm{d}}+n_{\mathrm{p}}$ multivariate neural network regression models

$$\left(\begin{array}{c}{\widehat{\Delta A}}_{\mathrm{d}}\left(\mu \right)\\ {\widehat{\Delta A}}_{\mathrm{p}}\left(\mu \right)\end{array}\right)=\left(\begin{array}{c}{\mathcal{R}}_{\mathrm{d},1}\left(\left(\begin{array}{c}{e_1^{n_{\mathrm{d}}}}^T({\tilde{A}}_{\mathrm{d}}\left(\mu \right)-{\overline{A}}_{\mathrm{d}}\left(\mu \right))\\ {e_1^{n_{\mathrm{d}}}}^T{\tilde{A}}_{\mathrm{d}}\left(\mu \right)\end{array}\right),{\theta }_{\mathrm{d},1}\right)\\ ...\\ {\mathcal{R}}_{\mathrm{d},n_{\mathrm{d}}}\left(\left(\begin{array}{c}{e_{n_{\mathrm{d}}}^{n_{\mathrm{d}}}}^T({\tilde{A}}_{\mathrm{d}}\left(\mu \right)-{\overline{A}}_{\mathrm{d}}\left(\mu \right))\\ {e_{n_{\mathrm{d}}}^{n_{\mathrm{d}}}}^T{\tilde{A}}_{\mathrm{d}}\left(\mu \right)\end{array}\right),{\theta }_{\mathrm{d},n_{\mathrm{d}}}\right)\\ {\mathcal{R}}_{\mathrm{p},1}\left(\left(\begin{array}{c}{e_1^{n_{\mathrm{p}}}}^T({\tilde{A}}_{\mathrm{p}}\left(\mu \right)-{\overline{A}}_{\mathrm{p}}\left(\mu \right))\\ {e_1^{n_{\mathrm{p}}}}^T{\tilde{A}}_{\mathrm{p}}\left(\mu \right)\end{array}\right),{\theta }_{\mathrm{p},1}\right)\\ ...\\ {\mathcal{R}}_{\mathrm{p},n_{\mathrm{p}}}\left(\left(\begin{array}{c}{e_{n_{\mathrm{p}}}^{n_{\mathrm{p}}}}^T({\tilde{A}}_{\mathrm{p}}\left(\mu \right)-{\overline{A}}_{\mathrm{p}}\left(\mu \right))\\ {e_{n_{\mathrm{p}}}^{n_{\mathrm{p}}}}^T{\tilde{A}}_{\mathrm{p}}\left(\mu \right)\end{array}\right),{\theta }_{\mathrm{p},n_{\mathrm{p}}}\right)\end{array}\right)$$

(34)

where $e_j^m$ denotes the jth canonical vector of ${\mathbb{R}}^m$.

Each of the regression ${\mathcal{R}}_{\mathrm{l},i}$ is a single-hidden layer bootstrap-aggregated neural network model with $n_{\mathrm{n}}$ neurons and $n_{\mathrm{nbs}}$ bootstrap replicates.

$${\mathcal{R}}_{\mathrm{l},i}\left(\left(\begin{array}{c}x\\ y\end{array}\right),{\theta }_{\mathrm{l},i}\right)=\frac{1}{k}\sum _{k=1}^{n_{\mathrm{nbs}}}\sum _{j=1}^{n_{\mathrm{n}}}{a}_j^{\left(k\right)}tanh\left(a_{\mathrm{x},j}^{\left(k\right)}x+a_{\mathrm{y},j}^{\left(k\right)}y+o_j^{\left(k\right)}\right)+o^{\left(k\right)}$$

(35)

Training

We sample artificial “data” by running the overkill computational model $n_{\mathrm{ml}}$ times, after sampling the training set parameters $\{{\mu }_1^{\mathrm{ml}},{\mu }_2^{\mathrm{ml}},...,{\mu }_{n_{\mathrm{ml}}}^{\mathrm{ml}}\}$ from prior ${\pi }_{\mu }$. The number of neurons and the cardinality of the training set is chosen automatically by making use of an automatised early-stopping methodology that aims to maximise the predictive coefficient of determination. We will not detail this procedure here. Fitting of the nonlinear regression coefficients is performed by employing standard least-squares method, solved by a gradient descent algorithm with randomised initialisation. Outliers of the set of bootstrap replicates are identified and eliminated to decrease the variance of the boostrap-aggregated regression model.

An example of fitted regression model is represented in Figure 6, where the output is the discretisation error in the first free vibration circular frequency (i.e., regression model ${\mathcal{R}}_{\mathrm{d},1}$).

5.4. Bootstrap Confidence Intervals for the MCMC Sampler

At any iteration n of the MCMC sampler, a Monte-Carlo estimate of the discretisation error for the posterior density of the ith component of $\mu$ is given by

$${\mathcal{D}}_{\mathrm{ks}}^{\left(\mathrm{n}\right)}:={\mathcal{D}}_{\mathrm{ks}}({\widehat{\pi }}_{{\mu }_i|d}^{\left(\mathrm{n}\right)}\left({\mu }_i\right),{\overline{\pi }}_{{\mu }_i|d}^{\left(\mathrm{n}\right)}\left({\mu }_i\right))=\underset{{\mu }_i}{sup}\left(\left|({\widehat{\Pi }}_{{\mu }_i|d}^{\left(\mathrm{n}\right)}\left({\mu }_i\right)-{\overline{\Pi }}_{{\mu }_i|d}^{\left(\mathrm{n}\right)}\left({\mu }_i\right))\right|\right)$$

(36)

where

$${\overline{\Pi }}_{{\mu }_i|d=d}^{\left(\mathrm{n}\right)}\left({\mu }_i\right):=\frac{1}{\tilde{n}}\sum _{k=n-\tilde{n}+1}^n\mathbb{I}({\mu }_i;{\overline{\mu }}_{k,i}\le {\mu }_i),$$

(37)

and

$${\widehat{\Pi }}_{{\mu }_i|d=d}^{\left(\mathrm{n}\right)}\left({\mu }_i\right):=\frac{1}{\tilde{n}}\sum _{k=n-\tilde{n}+1}^n\mathbb{I}({\mu }_i;{\widehat{\mu }}_{k,i}\le {\mu }_i).$$

(38)

Crucially, ${\mathcal{D}}_{\mathrm{ks}}^{\left(\mathrm{n}\right)}$ is a random variable whose statistics, and in particular its bias and variance, strongly depend on the length of the Markov chain. Unfortunately, evaluating the convergence of any statistics provided by MCMC is difficult, due to the statistical dependency between successively drawn samples.

Following standard diagnostic approaches for MCMC samplers (e.g., the Gelman-Rubin convergence test [53,54]), we will run $n_{\mathrm{c}}\ge 10$ independent (tempered) MCMC chains in parallel and pool all the resulting samples, after discarding the first 25% of every individual chain as burn-in (see Figure 7 as a visual aid). The pooled samples at iteration n of the multiple-chain MCMC (MC${}^3$) algorithm are

$${\overline{\mathcal{S}}}^{\left(n\right)}=\prod _{i=1}^{n_{\mathrm{c}}}{\overline{\mathcal{S}}}_i^{\left(n\right)}{\widehat{\mathcal{S}}}^{\left(n\right)}=\prod _{i=1}^{n_{\mathrm{c}}}{\widehat{\mathcal{S}}}_i^{\left(n\right)}$$

(39)

where ∏ denotes the cartesian product. In the previous expression, the sample set from chain i (at ambient temperature) is

$${\overline{\mathcal{S}}}_i^{\left(n\right)}=\{{\overline{\mu }}_{i,n-\tilde{n}+1},\cdots ,{\overline{\mu }}_{i,n}\}{\widehat{\mathcal{S}}}_i^{\left(n\right)}=\{{\widehat{\mu }}_{i,n-\tilde{n}+1},\cdots ,{\widehat{\mu }}_{i,n}\}$$

(40)

${\mathcal{D}}_{\mathrm{ks}}^{\left(\mathrm{n}\right)}$ is now the pooled KS distance estimate provided by the MC${}^3$ algorithm (we will keep the same notation for the sake of simplicity). Formally, we simply replace Equation (37) by

$${\overline{\Pi }}_{{\mu }_i|d=d}^{\left(\mathrm{n}\right)}\left({\mu }_i\right):=\frac{1}{n_{\mathrm{c}}}\frac{1}{\tilde{n}}\sum _{l=1}^{n_{\mathrm{c}}}\sum _{k=n-\tilde{n}+1}^n\mathbb{I}({\mu }_i;{\overline{\mu }}_{l,k,i}\le {\mu }_i),$$

(41)

and perform a similar operation to define the pooled corrected empirical distribution.

The independence of the $n_{\mathrm{c}}$ MCMC chains allows us to compute confidence intervals for ${\mathcal{D}}_{\mathrm{ks}}^{\left(\mathrm{n}\right)}$ by making use of the non-parametric bootstrap. This is done by resampling ${\overline{\mathcal{S}}}^{\left(n\right)}$ and ${\widehat{\mathcal{S}}}^{\left(n\right)}$ with replacement, generating bootstrap replicates of the pooled sample sets ${\left\{{\overline{\mathcal{S}}}_k^{\left(n\right)}\right\}}_{k\in 〚1n_{\mathrm{bs}}〛}$ and ${\left\{{\widehat{\mathcal{S}}}_k^{\left(n\right)}\right\}}_{k\in 〚1n_{\mathrm{bs}}〛}$,

$${\overline{\mathcal{S}}}_k^{\left(n\right)}=\prod _{i=\in {\mathcal{B}}_k}{\overline{\mathcal{S}}}_i^{\left(n\right)}{\widehat{\mathcal{S}}}_k^{\left(n\right)}=\prod _{i=\in {\mathcal{B}}_k}{\widehat{\mathcal{S}}}_i^{\left(n\right)},$$

(42)

where ${\mathcal{B}}_k\in {\left(〚1n_{\mathrm{c}}〛\right)}^{n_{\mathrm{bs}}}$ is such that each element of this set is drawn uniformly over $〚1n_{\mathrm{c}}〛$ and k varies between 1 and a large number $n_{\mathrm{bs}}$, typically set to 1000. For each replicate, statistics ${\mathcal{D}}_{\mathrm{ks}}^{\left(\mathrm{n}\right)}$, denoted by ${\mathcal{D}}_{\mathrm{ks},k}^{\left(\mathrm{n}\right)}$, can be computed in a straightforward manner by using the bootstrap replicates of the pooled empirical distributions

$${\overline{\Pi }}_{{\mu }_i|d=d,k}^{\left(\mathrm{n}\right)}\left({\mu }_i\right):=\frac{1}{n_{\mathrm{c}}}\frac{1}{\tilde{n}}\sum _{{\overline{\mu }}_i^{★}\in {\overline{\mathcal{S}}}_k^{\left(n\right)}}\mathbb{I}({\mu }_i;{\overline{\mu }}_i^{★}\le {\mu }_i),$$

(43)

$${\widehat{\Pi }}_{{\mu }_i|d=d,k}^{\left(\mathrm{n}\right)}\left({\mu }_i\right):=\frac{1}{n_{\mathrm{c}}}\frac{1}{\tilde{n}}\sum _{{\widehat{\mu }}_i^{★}\in {\widehat{\mathcal{S}}}_k^{\left(n\right)}}\mathbb{I}({\mu }_i;{\widehat{\mu }}_i^{★}\le {\mu }_i),$$

(44)

which reads as

$${\mathcal{D}}_{\mathrm{ks},k}^{\left(\mathrm{n}\right)}:={\mathcal{D}}_{\mathrm{ks}}({\widehat{\pi }}_{{\mu }_i|d,k}^{\left(\mathrm{n}\right)}\left({\mu }_i\right),{\overline{\pi }}_{{\mu }_i|d,k}^{\left(\mathrm{n}\right)}\left({\mu }_i\right))=\underset{{\mu }_i}{sup}\left(\left|({\widehat{\Pi }}_{{\mu }_i|d,k}^{\left(\mathrm{n}\right)}\left({\mu }_i\right)-{\overline{\Pi }}_{{\mu }_i|d,k}^{\left(\mathrm{n}\right)}\left({\mu }_i\right))\right|\right),$$

(45)

Finally, the bootstrap confidence intervals are calculated by calculating the Xth and $(100-X)$th bootstrap percentiles such that the Xth percentile reads as

$$q_X^{\left(\mathrm{n}\right)}=\left(Q_X\left({\left\{{\mathcal{D}}_{\mathrm{ks},k}^{\left(\mathrm{n}\right)}\right\}}_{k\in 〚1n_{\mathrm{bs}}〛}\right)-\mathrm{median}\left({\left\{{\mathcal{D}}_{\mathrm{ks},k}^{\left(\mathrm{n}\right)}\right\}}_{k\in 〚1n_{\mathrm{bs}}〛}\right)\right)+{\mathcal{D}}_{\mathrm{ks}}^{\left(\mathrm{n}\right)}$$

(46)

where $Q_X$ is an operator that extracts the Xth and Yth percentile of the set passed as argument.

It is important to understand that the derived bootstrap confidence interval stands for a chain of finite length n, and not for the asymptotic limit. In fact, estimate ${\mathcal{D}}_{\mathrm{ks}}^{\left(n\right)}$ of ${\mathcal{D}}_{\mathrm{ks}}$ is strongly biased (upward for small asymptotic errors ${\mathcal{D}}_{\mathrm{ks}}$ and typically downward for large asymptotic errors), which is due to two factors:

• the existence of the burn-in phase. For small n, each individual chain will be strongly affected by the initialisation of the chains. For small asymptotic errors, this can be expected to have a strong upward bias effect providing that the initialisation is disperse, which is often the case in practice (e.g., initialisation from the prior, sequential MC approaches with decreasing levels of noise). This can be visualised in Figure 7. Two incomplete chains running on the same probability density may be exploring completely different regions of space, yielding values of ${\mathcal{D}}_{\mathrm{ks}}$ that are large, even in the case where the corrected and uncorrected densities are close to one another.
• the discrete evaluation of the KS distance itself, which generates an additional (upward) bias.

The variance of ${\mathcal{D}}_{\mathrm{ks}}^{\left(n\right)}$ decreases with the number of chains of the MC${}^3$ algorithm (which is not a free parameter as overall CPU cost increases linearly with $n_{\mathrm{c}}$). Of course, bias and variance are both expected to decrease with the length n of the run.

5.5. Simultaneous Control of All Sources of Errors

We now make use of the CI derived for ${\mathcal{D}}_{\mathrm{ks}}^{\left(n\right)}$ to construct an adaptive inverse problem solver that jointly controls the quality of the mesh, and that of the statistical evaluation of the posterior densities. The algorithm is as follows. Given a current mesh ${\mathcal{M}}_i$ and number of iterations n of the Monte-Carlo solver, do:

• If $n=0$, perform $m_0$ iterations of the MC${}^3$ algorithm and set $n\leftarrow n+m_0$.
• Evaluate mesh convergence criterion ${\mathcal{C}}_{\mathrm{m}}$. If this criterion is satisfied, exit the adaptation procedure;
• If ${\mathcal{C}}_{\mathrm{m}}$ is not satisfied, evaluate statistical convergence criterion ${\mathcal{C}}_{\mathrm{s}}$

  -

  If ${\mathcal{C}}_{\mathrm{s}}$ is satisfied, set $i\leftarrow i+1$, reinitialise the MC${}^3$ algorithm and set $n=0$;

  -

  otherwise perform $m\left(n\right)$ iterations of the MC${}^3$ algorithm and set $n\leftarrow n+m\left(n\right)$ (m should be an exponentially increasing function).

5.5.1. Criterion ${\mathcal{C}}_{\mathrm{m}}$

Convergence of the finite element-based Bayesian inverse problem is achieved when ${\mathcal{D}}_{\mathrm{ks}}\le \gamma$, where $\gamma$ is a numerical tolerance that will typically be chosen between $0.01$ and $0.2$. As we only have Monte-Carlo estimates of ${\mathcal{D}}_{\mathrm{ks}}$, we require instead that

$${\mathcal{C}}_{\mathrm{m}}:q_{100-X}^{\left(\mathrm{n}\right)}\le \gamma$$

(47)

Criterion ${\mathcal{C}}_{\mathrm{m}}$ is an indicator of the combined effect of the discretisation and statistical errors onto the posterior densities. The discretisation error is evaluated through the choice of measure ${\mathcal{D}}_{\mathrm{ks}}$ as reliability indicator, whilst the statistical error is taken into account by making use of the upper limit of the bootstrap confidence interval for ${\mathcal{D}}_{\mathrm{ks}}^{\left(n\right)}$. The risk of falsely detecting mesh convergence due to a high statistical error is small, due to the fact that ${\mathcal{D}}_{\mathrm{ks}}^{\left(n\right)}$ is an upwardly biased estimate of ${\mathcal{D}}_{\mathrm{ks}}$ for small ${\mathcal{D}}_{\mathrm{ks}}$. Confidence in the result can be increased through a loose Gelman-Rubin convergence test in order to eliminate the risk of stopping the MC${}^3$ algorithm in its non-ergodic phase. However, this has proved to be unnecessary in our numerical tests. This is because ${\mathcal{C}}_{\mathrm{m}}$ is a rather strict criterion.

5.5.2. Criterion ${\mathcal{C}}_{\mathrm{s}}$

The second criterion will help us determine whether mesh refinement is actually needed, or whether the statistical error is too large for use to take a robust decision regarding mesh refinement. The ideal criterion is ${\mathcal{D}}_{\mathrm{ks}}\ge \eta$, where $\eta <\gamma$. The obvious strategy that would require $q_X^{\left(\mathrm{n}\right)}\le \eta$ does not work. This is due to the previously explained upward bias of estimator ${\mathcal{D}}_{\mathrm{ks}}^{\left(n\right)}$, which would eventually lead to the spurious satisfaction ${\mathcal{C}}_{\mathrm{s}}$, and consequently to systematic mesh refinement operations for low n count even when the mesh becomes fine enough for ${\mathcal{D}}_{\mathrm{ks}}$ to be well below target $\gamma$. In order to derive an appropriate criterion ${\mathcal{C}}_{\mathrm{s}}$, we remark that whilst the ideal criterion ${\mathcal{D}}_{\mathrm{ks}}\ge \eta$ cannot be statistically evaluated for general values of $\eta$, criterion ${\mathcal{D}}_{\mathrm{ks}}>0$ can be. More precisely, we propose to estimate the statistical error by evaluation whether, at current n, the corrected and uncorrected posterior distributions can be statistically distinguished.

We postulate the following null hypothesis: Hypothesis 0 (H0): the corrected and uncorrected posterior densities are identical.

The rejection of the null hypothesis will indicate that the two densities are significantly different. Now, the criterion for the need for mesh refinement becomes the following:

$${\mathcal{C}}_{\mathrm{s}}:\mathrm{Pr}\left({\mathcal{D}}_{\mathrm{ks}}^{\left(\mathrm{n}\right)}\ge q_Z^{\left(\mathrm{n}\right)}|\mathrm{H}0\right)\le \xi$$

(48)

The probability density of ${\mathcal{D}}_{\mathrm{ks}}^{\left(\mathrm{n}\right)}$ under null hypothesis H0 can be approximated by adapting the previously described bootstrap procedure. We will estimate the density of

$${\mathcal{D}}_{\mathrm{ks},0}^{\left(\mathrm{n}\right)}:={\mathcal{D}}_{\mathrm{ks}}({\overline{\pi }}_{{\mu }_i|d}^{\left(\mathrm{n}\right),1}\left({\mu }_i\right),{\overline{\pi }}_{{\mu }_i|d}^{\left(\mathrm{n}\right),2}\left({\mu }_i\right))=\underset{{\mu }_i}{sup}\left(\left|({\overline{\Pi }}_{{\mu }_i|d}^{\left(\mathrm{n}\right),1}\left({\mu }_i\right)-{\overline{\Pi }}_{{\mu }_i|d}^{\left(\mathrm{n}\right),2}\left({\mu }_i\right))\right|\right)$$

(49)

where ${\overline{\pi }}_{{\mu }_i|d}^{\left(\mathrm{n}\right),1}$ and ${\overline{\pi }}_{{\mu }_i|d}^{\left(\mathrm{n}\right),2}$ are obtained, respectively, by pooling the result of two independent runs of $n_{\mathrm{c}}$ Markov chains of lengths n and corresponding to the uncorrected finite element model only (one can equally choose the corrected one). Notice, to clarify the idea, that ${\mathcal{D}}_{\mathrm{ks},0}^{\left(\mathrm{n}\right)}$ tends to 0 as n tends to infinity: this is a measure of the statistical error only. The desired density can be estimated by resampling ${\overline{\mathcal{S}}}^{\left(n\right)}$ twice, computing the KS distance between the two pooled sets of samples, and repeating the operation $n_{\mathrm{bs}}$ times, which generates a sequence of real numbers $\left\{{\mathcal{D}}_{\mathrm{ks},0,k}^{\left(\mathrm{n}\right)}\right\}$. The replicated empirical cumulative distributions are

$${\overline{\Pi }}_{{\mu }_i|d=d,k}^{\left(\mathrm{n}\right),\mathrm{j}}\left({\mu }_i\right):=\frac{1}{n_{\mathrm{c}}}\frac{1}{\tilde{n}}\sum _{{\overline{\mu }}_i^{★}\in {\overline{\mathcal{S}}}_k^{\left(n\right),j}}\mathbb{I}({\mu }_i;{\overline{\mu }}_i^{★}\le {\mu }_i)$$

(50)

with the sampling sets

$${\overline{\mathcal{S}}}_k^{\left(n\right),j}=\prod _{i=\in {\mathcal{B}}_k^j}{\overline{\mathcal{S}}}_i^{\left(n\right)}$$

(51)

where $j=1$ or $j=2$ and ${\mathcal{B}}_k^1$ and ${\mathcal{B}}_k^2$ are sets of $〚1n_{\mathrm{c}}{〛}^{n_{\mathrm{c}}}$ constructed such that each element of these two sets is drawn uniformly over $〚1n_{\mathrm{c}}〛$. We can now extract the $(100\times (1-\xi \left)\right)$th percentile $q_{Y,0}^{\left(\mathrm{n}\right)}$ corresponding to p-value $\xi$, and evaluate whether $q_{Y,0}^{\left(\mathrm{n}\right)}\le q_Y^{\left(\mathrm{n}\right)}$. If so, test ${\mathcal{C}}_{\mathrm{s}}$ is true. If not, the statistical sampling error is too large to allow us to decide whether mesh refinement should be performed or not. In this case, and assuming that mesh convergence criterion ${\mathcal{C}}_{\mathrm{m}}$ is not satisfied, we need to continue sampling with MCMC.

Notice that the use of a larger percentile Z yields a less conservative indicator for the need for mesh refinement, which can be compensated by decreasing p-value $\xi$. The criterion that is arguably the easiest to interpret is $\mathrm{Pr}\left({\mathcal{D}}_{\mathrm{ks}}^{\left(\mathrm{n}\right)}\ge q_{50}^{\left(\mathrm{n}\right)}|\mathrm{H}0\right)\le \xi$, where $q_{50}^{\left(\mathrm{n}\right)}$ is directly the KS distance ${\mathcal{D}}_{\mathrm{ks}}^{\left(\mathrm{n}\right)}$ (i.e., computed without bootstrapping).

6. Numerical Examples—Part II. Automatised Error Control and Discussion

We now come back to the numerical examples introduced in Section 3 and produce three series of results, illustrating three difference aspects of the proposed approach.

6.1. MCMC Iterations Only When Needed

The results reported in Figure 8 correspond to Test 1. For now, we aim to control the quality of the posterior density of the first parameter ${\mu }_1$. Each of the graphs shows the evolution of various statistics of the corresponding KS distance as a function of the number of iteration n of the tempered MC${}^3$ algorithm. The mesh used is displayed on the left-hand side of each of the graphs. The solid black line represents the evolution ${\mathcal{D}}_{\mathrm{ks}}^{\left(n\right)}$ itself. The 5th and and 95th bootstrap percentiles of this quantity are also reported, in solid blue line and solid red line, respectively. We set convergence criterion ${\mathcal{C}}_{\mathrm{m}}$ such that $\gamma =0.2$ and $Y=95$ for the pure discretisation error. Therefore, convergence with respect to the finite element discretisation is obtained when, for sufficient long chains, the red line (upper limit of the bootstrap CI for ${\mathcal{D}}_{\mathrm{ks}}^{\left(n\right)}$) is below $\gamma$.

In grey colours, we report evolution of ${\mathcal{D}}_{\mathrm{ks},0}^{\left(n\right)}$ for the coarse scale (light grey), consistently with the derivations of the previous section, but also, for reference, a similar statistics constructed with the corrected finite element model (dark grey). Both curves have similar evolutions and, of course, tend to 0 as n tends to infinity. We set $Z=5$, and $\eta =0.05$. This means that the statistical error is considered small enough once the blue line (lower limit of the bootstrap CI for ${\mathcal{D}}_{\mathrm{ks}}^{\left(n\right)}$) is above the 95th percentile of ${\mathcal{D}}_{\mathrm{ks},0}^{\left(n\right)}$. Here, “small enough” is to be understood, as small enough for a decision regarding the need to refine the mesh one step further to be taken.

The results are as follows. For the coarsest mesh, we detect a separation of the discretisation and statistical errors at iteration 60 (vertical solid red line). ${\mathcal{C}}_{\mathrm{s}}$ is satisfied. As ${\mathcal{C}}_{\mathrm{m}}$ is not satisfied, we can stop the MCMC sampler and refine the mesh. A similar behaviour is seen for mesh ${\mathcal{M}}_2$. For the last mesh, convergence is achieved after 530 iterations, ${\mathcal{C}}_{\mathrm{s}}$ never reaching satisfaction. The discretisation and sampling errors remain entangled, but the sum of them is below the desired target.

We can see here that the proposed algorithm allows us to stop the MCMC sampler early, moving directly to the level of mesh refinement that will yield the desired quality of the posterior densities.

6.2. Goal-Oriented Error Control

The second set of results correspond to Test 1, still. However, we now monitor the convergence of the posterior predictive density of the fourth eigenvalue. This is reported in the top two graphs of Figure 9. For ${\mathcal{M}}_1$ Early-stopping is performed at iteration 40 of the MCMC sampler. For ${\mathcal{M}}_2$, global convergence is achieved after 120 iterations, and this error keeps decreasing. The MCMC is allowed to continue generating samples, meaning that the discretisation error is actually a lot smaller than the requested tolerance $\gamma$. It is interesting to see that the convergence is much faster than for the first parameter, whose convergence was studied in the previous subsection. This shows that for the same inverse problem, the algorithm may spend more or less resources depending on the engineering quantity of interest.

6.3. Uncertainty-Driven Error Control

Finally, the last set of results concerns Test 2. Here, remember that the three first eigenvalues are used as measurements. We control the convergence of the fourth eigenvalue, which was controlled in the context of Test 1 in the previous subsection. We see here that for mesh ${\mathcal{M}}_2$, the posterior density of the fourth eigenvalue is still far from convergence, whilst it was evaluated in a very precise way in Test 1. This shows that the proposed error control algorithm automatically adapts to the level of posterior uncertainty. Qualitatively, a wide posterior density is associated with a high uncertainty concerning the value of QoIs. Consequently, the mesh does not need to be very refined to capture the posterior density correctly. Conversely, for Test 2, the posterior density is sharper, and similar levels of discretisation errors have a much stronger impact on the KS distance.

7. Concluding Remarks and Discussion

We have presented a methodology to control the various sources of errors arising in finite-element based Bayesian inverse problems. We have focussed on a simple numerical approximation chain consisting of (i) a finite element discretisation of the continuum mechanics problem and (ii) a MCMC solver to draw samples from posterior density distributions. So far we did not consider further error sources such as that engendered by meta-modelling. We have showed that it was possible to drive the mesh refinement process in a goal-oriented manner, by quantifying the impact of the associated error onto posterior density distributions. In order to do so, we run two independent MCMC simultaneously, one of them using a corrected likelihood function that takes the discretisation error. Any a posteriori error estimate available for the PDE under consideration may be used to obtain this correction. Of course, the accuracy (effectivity) of the chosen error estimate will impact the accuracy of the methodology developed in this paper. The study and control of this effect will require further research to be carried out,

We have shown that by using multiple replicates of the component-wise MCMC, it was also possible to (i) derive bootstrap-based confidence intervals for the error in posterior densities engendered by the spatial discretisation of the underlying PDE. Importantly, this approach allows us to stop the MCMC iteration as soon as we can be sufficiently confident in the fact that the posterior densities obtained with the current mesh are not accurate enough for our purpose. The approach is goal-oriented: the adaptivity is performed in the sense of the posterior distribution of either some of the latent parameters, or in the sense of the predictive posterior density of engineering QoIs. Finally, we have shown that the approach is uncertainty driven: it will only refine the mesh and/or request additional samples to be drawn by the MCMC algorithm if the effect on the QoIs can be felt when measured using a statistical distance between their posterior densities. In particular, model updating problems with wider priors tend to require less computational effort than parameter estimation problems with very rich observed data and/or narrow prior densities. The proposed methodology establishes a bridge between the field of reliability estimation for the sampling-based algorithms used to solve probabilistic inverse problems, and the field of deterministic, goal-oriented finite element error estimation.

Although the results presented in this paper are encouraging, the proposed approach has shortcomings that need to be addressed in future research work. The error estimation procedure is relatively wasteful for two reasons. Firstly, multiple MCMC algorithms need to run independently for bootstrapping to be possible. These chains all exhibit their own burn-in phase, which increases the amount of discarded samples. Secondly, the finite element error estimation procedure is not merely a post-processing operation any longer; one cannot post-process the finite element results corresponding to the uncorrected chain in order to compute the corrected likelihood. This is because both chains are independent and require running the coarse finite element simulations at different points of the parameter domain. Finally, let us acknowledge that Bayesian finite element inverse problems should not be solved by a MCMC solver without constructing a surrogate model first. The number of finite element computations involved, even if the proposal distribution is well designed, is in the tens of thousands. We are currently investigating the use of Polynomial Chaos surrogates, which adds another layer of numerical approximation that needs to be controlled in a robust and efficient manner. In this context, an elegant approach to separate the sources of errors (i.e., Finite Element discretisation error and Polynomial Chaos error) may be found in [55], which may constitute a solid starting point for the next step of our investigations.