ANOVA-GP Modeling for High-Dimensional Bayesian Inverse Problems

Abstract

Markov chain Monte Carlo (MCMC) stands out as an effective method for tackling Bayesian inverse problems. However, when dealing with computationally expensive forward models and high-dimensional parameter spaces, the challenge of repeated sampling becomes pronounced. A common strategy to address this challenge is to construct an inexpensive surrogate of the forward model, which cuts the computational cost of individual samples. While the Gaussian process (GP) is widely used as a surrogate modeling strategy, its applicability can be limited when dealing with high-dimensional input or output spaces. This paper presents a novel approach that combines the analysis of variance (ANOVA) decomposition method with Gaussian process regression to handle high-dimensional Bayesian inverse problems. Initially, the ANOVA method is employed to reduce the dimension of the parameter space, which decomposes the original high-dimensional problem into several low-dimensional sub-problems. Subsequently, principal component analysis (PCA) is utilized to reduce the dimension of the output space on each sub-problem. Finally, a Gaussian process model with a low-dimensional input and output is constructed for each sub-problem. In addition to this methodology, an adaptive ANOVA-GP-MCMC algorithm is proposed, which further enhances the adaptability and efficiency of the method in the Bayesian inversion setting. The accuracy and computational efficiency of the proposed approach are validated through numerical experiments. This innovative integration of ANOVA and Gaussian processes provides a promising solution to address challenges associated with high-dimensional parameter spaces and computationally expensive forward models in Bayesian inference.

1. Introduction

In numerous scientific and engineering domains, the inference of parameters of interest through observations, such as inverse problems [1], is a common requirement. For instance, in seismology, inverse problems arise in the attempt to determine the subsurface properties of the Earth, such as seismic velocity or material composition, based on recorded seismic waves from earthquakes or controlled sources [2]. Another example is heat conduction, which involves determining the initial temperature distribution or the thermal properties of a material from given temperature measurements at certain locations and times [3,4]. Recently, a statistical model for system reliability evaluation by jointly considering the correlated component lifetimes and the lifetime ordering constraints has been proposed in [5]. Based on the proposed model, the estimation of model parameters from the observed data is also discussed. In [6], two semiparametric additive mean models are proposed for clustered panel count data, and some estimation equations are derived to estimate the regression parameters of interest for the proposed two models. In this study, we focus on inverse problems.

Classical inference methods typically yield a single-point estimate of the parameters by optimizing an objective function with a regularization term. However, Bayesian inference methods offer the advantage of not only providing parameter estimates but also quantifying the uncertainty associated with the obtained results. This characteristic proves more practical for addressing inverse problems [7]. Bayesian methods treat parameters as random variables. They establish prior distributions for these variables based on existing knowledge and then calculate the conditional probability, also known as the posterior distribution of the parameters, given the observations using Bayes’ theorem. Consequently, statistical information such as the expectation and variance of the parameters can be derived.

While the concept of Bayesian inference methods is straightforward, practical implementation can encounter significant challenges. In most instances, the posterior probability distribution lacks an explicit expression and can only be approximated through numerical methods. Typically, the Markov chain Monte Carlo (MCMC) method [8] is employed to generate samples from the posterior distribution, which enables the estimation of posterior statistical information based on these samples. It is crucial to note that the MCMC method necessitates repeated evaluations of the forward model: entailing the complete simulation of the mapping from parameters to observations. However, the forward problem often involves the solution of computationally intensive models such as partial differential equations, especially for practical problems like groundwater inverse modeling [9] and seismic inversion [10]. Furthermore, achieving a robust posterior estimate typically requires a substantial number of samples. These two factors render MCMC simulations impractical for large-scale realistic problems.

To enhance the computational efficiency of MCMC simulations, two primary approaches can be pursued. The first involves reducing the number of required samples, which necessitates the development of more efficient sampling methods. The second aims to minimize the computational cost of a single sample. This can be achieved by pre-building a low-cost surrogate or simplified model for use in the MCMC process, which aligns with the focus of our study.

Existing research methods include polynomial chaos expansion [11], the Gaussian process [12,13], sparse grid interpolation [14], and reduced-basis models [15,16]. The performance of these methods for accelerating Bayesian computation is discussed in [17]. While successful for various inverse problems, their applicability is often constrained by the high dimensionality of unknown parameters in practical scenarios. For instance, hydraulic conductivity in groundwater models and wave speed in seismic inversion, which are often characterized by random fields, can result in dimensions ranging from tens of thousands to even more. Although dimension-reduction methods like truncated Karhunen–Loève (KL) expansion can be employed [18], practical parameters are typically nonsmooth and thus require a substantial number of KL modes. Directly building surrogates for such high-dimensional problems poses a significant challenge. The analysis of variance (ANOVA) methods [19,20], which are proposed for efficiently solving high-dimensional problems, aim to decompose a high-dimensional parameter space into a union of low-dimensional spaces such that standard surrogate modeling strategies can be applied. The adaptive reduced basis ANOVA method combines ANOVA decomposition with reduced basis methods [21,22] to significantly reduce the computational cost of Bayesian inverse problems without sacrificing accuracy. The combination of ANOVA decomposition with the Gaussian process (GP) is studied in various fields, such as designing the surrogate model for the forward problem [13] and recovering functional dependence from sparse data [23,24]. In this study, we primarily concentrate on ANOVA-GP modeling for high-dimensional Bayesian inverse problems.

In this study, we introduce a novel approach that integrates ANOVA decomposition with Gaussian processes to construct surrogate models. The resulting surrogate model is seamlessly integrated into MCMC simulations, which provides a substantial acceleration in computation. This innovative approach addresses the challenges posed by high-dimensional input and output spaces and holds promise for efficiently handling complex problems in Bayesian inference. To further enhance the efficiency, we propose an adaptive ANOVA-GP-MCMC algorithm that is designed to dynamically update the ANOVA-GP model based on posterior samples obtained during the MCMC process. Numerical results demonstrate the efficiency and effectiveness of the algorithm for quantifying uncertainties of the parameters.

The remainder of the paper is structured as follows. Section 2 elaborates the problem formulation considered in this study, which encompasses Bayesian inverse problems and partial differential equations with random parameters. In Section 3, we delve into the methodology for constructing the ANOVA-GP model and introduce the adaptive ANOVA-GP-MCMC algorithm. Section 4 is dedicated to validating the accuracy and computational efficiency of the proposed approach for high-dimensional inverse problems through a series of numerical experiments. Finally, Section 5 encapsulates a summary of our research and outlines prospects for future work.

2. Problem Setup

2.1. Bayesian Inverse Problem

Let $\mathcal{X}$ and $\mathcal{Y}$ denote two separable Banach spaces and consider a forward model G mapping from $\mathcal{X}$ to $\mathcal{Y}$. Assume that for an unknown parameter vector $\xi \in \mathcal{X}$, the only available observation is $d=\mathfrak{e}\left(G\right(\xi \left)\right)$, where $\mathfrak{e}:\mathcal{Y}\to \mathcal{Y}$ introduces measurement noise. The inverse problem involves inferring the parameter $\xi$ from the observed data d.

In classical statistical inference, parameters are traditionally treated as deterministic values, and a common approach involves obtaining a point estimate by maximizing the likelihood function. However, in a Bayesian framework, parameters are treated as random variables and are assigned a prior distribution to encapsulate prior knowledge. Upon making an observation, belief in the parameter is updated using Bayes’ rule: leading to the derivation of the posterior distribution. This Bayesian approach not only yields a single estimate but also provides a quantification of uncertainty associated with the parameter.

Let $\pi \left(\xi \right):\mathcal{X}\to {\mathbb{R}}_0^{+}$ denote the prior probability density function of the parameter and satisfy ${\int }_{\mathcal{X}}\pi \left(\xi \right)\mathrm{d}\xi =1$. This prior probability reflects our belief in the parameter before any observations are made and can be suitably chosen based on historical data, model structures, and expert knowledge.

Consider $L(d,\xi ):\mathcal{Y}\times \mathcal{X}\to {\mathbb{R}}_0^{+}$ as the likelihood function, which represents the probability of the observed value d conditional on the parameter $\xi$. According to Bayes’ rule, the posterior probability density function of the parameter is expressed as

$$\pi \left(\xi \right|d)=\frac{L(d,\xi )\pi \left(\xi \right)}{{\int }_{\mathcal{X}}L(d,\xi )\pi \left(\xi \right)\mathrm{d}\xi }.$$

(1)

The denominator ${\int }_{\mathcal{X}}L(d,\xi )\pi \left(\xi \right)\mathrm{d}\xi$ is referred to as the evidence or normalization constant and equals the marginal probability of the observed data.

The likelihood function is constructed by assessing the disparity between the observed data vector d and the output of the forward model $G\left(\xi \right)$. In many practical physical processes, an additive error model is commonly employed; it is represented as $d=G\left(\xi \right)+ϵ$, where $ϵ$ accounts for measurement errors. Moreover, it is often assumed that $ϵ$ is unbiased and uncorrelated across dimensions. Each dimension of $ϵ$ follows a normal distribution with zero mean and variance ${\sigma }^2$: denoted as ${\pi }_{ϵ}\sim \mathcal{N}(0,{\sigma }^{2}I)$. Under these assumptions, the likelihood function can be expressed as

$$L(d,\xi )={\pi }_{ϵ}(d-G\left(\xi \right))=\exp \left\{\frac{{\parallel d-G\left(\xi \right)\parallel }_2^2}{2{\sigma }^2}\right\},$$

(2)

thus we have

$$\pi \left(\xi \right|d)\propto exp\left\{\frac{{\parallel d-G\left(\xi \right)\parallel }_2^2}{2{\sigma }^2}+log\pi \left(\xi \right)\right\}.$$

(3)

While for the aforementioned assumptions the posterior density function $\pi \left(\xi \right|d)$ is known, there are two challenges that hinder the derivation of a straightforward explicit expression. The first challenge arises when the forward model functions as a black box or lacks an attainable explicit expression. The second challenge involves the computation of the normalizing constant, which may require handling intricate high-dimensional integrals that are either computationally demanding or, in some cases, infeasible. Hence, in this study, Markov chain Monte Carlo (MCMC) is employed to generate samples conforming to the posterior distribution. These samples are then used for estimating both the expected value and the uncertainty associated with the parameters.

MCMC was initially introduced by Metropolis [25] and later extended by Hastings [26]. This method constructs a Markov chain with the posterior distribution as the stationary distribution through a random walk. In our investigation, we employ the classic Metropolis–Hastings (MH) algorithm (refer to Algorithm 1) to generate a set of N samples conforming to the posterior distribution of the parameters. The proposal distribution is denoted as ${\pi }_p(·|{\xi }^{\left(j\right)})$ and is commonly chosen as a multivariate normal distribution with mean ${\xi }^{\left(j\right)}$.

Algorithm 1 Classic Metropolis–Hastings (MH) algorithm

Input: Number of samples N, forward model G, noise distribution ${\pi }_{ϵ}$, proposal distribution ${\pi }_p$, prior distribution $\pi$, and observation d. Output: Posterior samples ${\left\{{\xi }^{\left(j\right)}\right\}}_{j=1}^N$. 1: Draw a sample ${\xi }^{\left(0\right)}$ from the prior distribution $\pi$ as the initial state. 2: for $j:=1:N$ do 3:     Generate the proposal ${\xi }^{*}\sim {\pi }_p(·|{\xi }^{(j-1)})$. 4:     Compute the acceptance rate $a:=min\left(1,\frac{{\pi }_{ϵ}(d-G\left({\xi }^{*}\right))\pi \left({\xi }^{*}\right)}{{\pi }_{ϵ}(d-G\left({\xi }^{(j-1)}\right))\pi \left({\xi }^{(j-1)}\right)}·\frac{{\pi }_p\left({\xi }^{(j-1)}\right|{\xi }^{*})}{{\pi }_p\left({\xi }^{*}\right|{\xi }^{(j-1)})}\right)$. 5:     Draw $\rho \sim \mathrm{Uniform}[0,1]$. 6:     if $\rho <a$ then 7:         Let ${\xi }^{\left(j\right)}={\xi }^{*}$. 8:     else 9:         Let ${\xi }^{\left(j\right)}={\xi }^{(j-1)}$. 10:    end if 11: end for

As can be seen from the algorithm, each iteration of MCMC involves an evaluation of the forward model, rendering the MCMC process time-consuming. In the subsequent sections, we will introduce a strategy to expedite MCMC computations by employing surrogate models constructed using the ANOVA-GP method.

2.2. Partial Differential Equations with Random Parameters

In this study, we address Bayesian inverse problems for which the forward model is governed by partial differential equations with high-dimensional random parameters [13,21,22]. Specifically, details of the forward model considered in this paper are addressed as follows. Let D denote a physical domain (a subset of ${\mathbb{R}}^2\mathrm{or}{\mathbb{R}}^3$) that is bounded and connected and has a polygonal boundary $\partial D$. Let $\xi ={[{\xi }_1,\cdots ,{\xi }_M]}^{\mathrm{T}}$ be a random vector of dimension M, and denote its image by $I^M$. The associated problem of the forward model can be formulated as finding a function $s(x,\xi ):D\times I^M\to \mathbb{R}$ that satisfies

$$\left\{\begin{array}{cc}\hfill \mathcal{L}(x,\xi ;s(x,\xi \left)\right)& =f\left(x\right),\forall (x,\xi )\in D\times I^M,\hfill \\ \hfill \mathfrak{b}(x,\xi ;s(x,\xi \left)\right)& =g\left(x\right),\forall (x,\xi )\in \partial D\times I^M,\hfill \end{array}\right.$$

(4)

where $\mathcal{L}$ is a partial differential operator, $\mathfrak{b}$ is a boundary operator, f denotes the source function, and g determines the boundary conditions of the system. With an observation operator $\mathfrak{c}$ (e.g., extracting values of the solution u at specific points in physical space), the forward model is expressed as

$$G\left(\xi \right):=u\left(\xi \right)=\mathfrak{c}\left(s\right(\xi \left)\right).$$

(5)

During the Markov chain Monte Carlo (MCMC) process, each forward model evaluation involves solving Equation (4). As mentioned earlier, our study will employ a computationally convenient surrogate model to avoid repeatedly solving the original equation, thereby enhancing the computational efficiency of MCMC.

3. Methodology

This section initiates with an elaborate description of the standard methodologies that form the foundation for our study. These methodologies include ANOVA decomposition, principal component analysis, and Gaussian process regression. Following this, we introduce the integration of these approaches to construct a surrogate model for the forward model discussed earlier: termed the ANOVA-GP model. Finally, we apply the ANOVA-GP model to MCMC sampling and propose an adaptive ANOVA-GP-MCMC approach. In this approach, the surrogate is dynamically updated based on the posterior knowledge acquired during the inversion process. This adaptive approach further enhances the efficiency and adaptability of the surrogate model in the context of Bayesian inference.

3.1. ANOVA Decomposition

The general process of ANOVA decomposition, as outlined in [13,20,22], can be described as follows. Assume $u\left(\xi \right)\in L^2\left(I^M\right)$, where $\xi ={[{\xi }_1,\cdots ·,{\xi }_M]}^{\mathrm{T}}\in I^M$. Let $\mathcal{P}$ be the collection of coordinate indices $\{1,\cdots ,M\}$, and any non-empty subset $t\subseteq \mathcal{P}$ is referred to as an ANOVA index, with its cardinality denoted by $\left|t\right|$. For any given t, ${\xi }_t$ represents a $\left|t\right|$-dimensional vector composed of the values of $\xi$ at corresponding indices. For instance, if $t=1,2,4$, then $\left|t\right|=3$ and ${\xi }_t={[{\xi }_1,{\xi }_2,{\xi }_4]}^{\mathrm{T}}\in I^3$. Let $\mathrm{d}\mu$ be a probability measure on $I^M$. The ANOVA decomposition of $u\left(\xi \right)$ can be expressed as the sum of $2^M$ terms:

$$u\left(\xi \right)=u_0+\sum _{t\ne \varnothing ,t\subseteq \mathcal{P}}{u}_t\left({\xi }_t\right),$$

(6)

where

$$u_0={\int }_{I^M}u\left(\xi \right)\mathrm{d}\mu \left(\xi \right),$$

(7)

and $u_t\left({\xi }_t\right)$ is recursively defined as

$$u_t\left({\xi }_t\right)={\int }_{I^{M-\left|t\right|}}u\left(\xi \right)\mathrm{d}\mu \left({\xi }_{\mathcal{P}\setminus t}\right)-\sum _{s\ne \varnothing ,s\subset t}{u}_s\left({\xi }_s\right)-u_0,\forall t\subseteq \mathcal{P},$$

(8)

where $u_t\left({\xi }_t\right)$ is referred to as a $\left|t\right|$-order ANOVA term. Since we assume that ${\left\{{\xi }_i\right\}}_{i=1}^M$ are independent of each other, $\mathrm{d}\mu \left({\xi }_{\mathcal{P}\setminus t}\right)={\prod }_{i\in \mathcal{P}\setminus t}\mathrm{d}\mu \left({\xi }_i\right)$.

For any high-dimensional function, the ANOVA decomposition is a finite and exact expansion [19,27]. Additionally, every term in the expansion is orthogonal to the other terms [20], i.e.,

$${\int }_{I^M}{u}_t\left({\xi }_t\right)u_w\left({\xi }_w\right)\mathrm{d}\mu \left(\xi \right)={\delta }_{tw},\forall t,w\subseteq \mathcal{P},$$

(9)

and thus, every term except $u_0$ has zero mean:

$${\int }_{I^M}{u}_t\left({\xi }_t\right)\mathrm{d}\mu \left(\xi \right)=0,\forall t\subseteq \mathcal{P}\mathrm{and}t\ne \varnothing .$$

(10)

By (9), the variance of $u\left(\xi \right)$ can be computed via

$$\mathbb{V}\left(u\left(\xi \right)\right)=\sum _{t\ne \varnothing ,t\subset \mathcal{P}}\mathbb{V}\left(u_t\left({\xi }_t\right)\right)$$

(11)

3.1.1. Anchored ANOVA Decomposition

When using the Lebesgue measure $\mathrm{d}\mu \left(\xi \right):=\mathrm{d}\xi$, Equation (6) is referred to as the classic ANOVA decomposition. However, this involves multiple high-dimensional integrals, necessitating complex computations. To alleviate this, one often utilizes the Dirac measure $\mathrm{d}\mu \left(\xi \right):=\delta (\xi -c)\mathrm{d}\xi ={\prod }_{i=1}^M\delta ({\xi }_i-c_i)\mathrm{d}{\xi }_i$, where $c={[c_1,\cdots ,c_M]}^{\mathrm{T}}\in I^M$ is a given anchor point. This approach is known as a Dirac ANOVA decomposition or anchored ANOVA decomposition [20,28]. Under this measure, Equations (7) and (8) can be expressed as

$$u_0=u\left(c\right),u_t\left({\xi }_t\right)=u\left({\xi }^{c,t}\right)-\sum _{s\ne \varnothing ,s\subset t}{u}_s\left({\xi }_s\right)-u_0,\forall t\subseteq \mathcal{P},$$

(12)

where ${\xi }^{c,t}={[{\xi }_1^{c,t},\cdots ,{\xi }_M^{c,t}]}^{\mathrm{T}}$ is defined as

$${\xi }_i^{c,t}=\left\{\begin{array}{cc}{c}_i,\hfill & \mathrm{for}i\in \mathcal{P}\setminus t\hfill \\ {\xi }_i,\hfill & \mathrm{for}i\in t\hfill \end{array}.\right.$$

(13)

This naturally raises a question: How should the anchor point c be chosen? Theoretically, any anchor point c can provide an accurate ANOVA decomposition. However, a judicious choice of the anchor point enables approximation of the original function with a small subset of expansion terms [20,28], which is a crucial factor for enhancing computational efficiency. In previous studies, selecting parameter points with function values close to the mean has proven to be effective. However, calculating the expected value of the function is usually challenging and can only be obtained through inexact numerical approximation. Another choice of the anchor point, as demonstrated in [20], is the expectation of the parameter, which is a strategy also adopted in our study.

3.1.2. Selection of the ANOVA Terms

In Equation (6), as the parameter dimension M increases, the number of terms in the ANOVA expansion grows exponentially, posing a significant challenge to computing the ANOVA decomposition for high-dimensional functions. In practice, it is often sufficient to retain only a small subset of the lower-order ANOVA terms to approximate the original function effectively [13,22]. Let $\mathcal{J}$ be the set of active ANOVA indices; the original function can then be approximated as

$$u\left(\xi \right)\approx u_{\mathcal{J}}\left(\xi \right)=u_0+\sum _{t\in \mathcal{J}}{u}_t\left({\xi }_t\right),$$

(14)

which is also called the truncated ANOVA expansion. In this work, we adopt the method for selecting active ANOVA terms as explored in [22]. Define the set of active ANOVA indices at each order i as

$${\mathcal{J}}_i:=\left\{t\right|t\in \mathcal{J}and\left|t\right|=i\};$$

(15)

then $\mathcal{J}={\cup }_{i=1}^M{\mathcal{J}}_i$. Initialize the set of first-order ANOVA indices as ${\mathcal{J}}_1=\left\{1,\cdots ,M\right\}$. Assuming the prior distribution ${\pi }_t\left({\xi }_t\right)$ of parameter ${\xi }_t$ is known, the prior expectation of $u_t\left({\xi }_t\right)$ is

$$\mathbb{E}\left(u_t\right):={\int }_{I^{\left|t\right|}}{u}_t\left({\xi }_t\right){\pi }_t\left({\xi }_t\right)\mathrm{d}{\xi }_t;$$

(16)

the relative expectation of each ANOVA term is defined as

$${\gamma }_t:=\frac{\parallel \mathbb{E}\left(u_t\right){\parallel }_2}{\parallel u_0+{\sum }_{s\in {\mathcal{J}}_1\cup \cdots \cup {\mathcal{J}}_{\left|t\right|-1}}\mathbb{E}\left(u_s\right){\parallel }_2},$$

(17)

where ${\parallel ·\parallel }_2$ represents the L2 norm of the function. The prior expectation of the ANOVA term $\mathbb{E}\left(u_t\right)$ can be estimated using the Monte Carlo method:

$$\tilde{\mathbb{E}}\left(u_t\right):=\frac{1}{N_{model}}\sum _{j=1}^{N_{model}}{u}_t\left({\xi }_t^{\left(j\right)}\right),$$

(18)

where ${\left\{{\xi }_t^{\left(j\right)}\right\}}_{j=1}^{N_{model}}$ are $N_{model}$ samples drawn from the prior distribution ${\pi }_t\left({\xi }_t\right)$; thus, the relative expectation can be approximated as

$$\widetilde{{\gamma }_t}:=\frac{\parallel \tilde{\mathbb{E}}\left(u_t\right){\parallel }_2}{\parallel u_0+{\sum }_{s\in {\mathcal{J}}_1\cup \cdots \cup {\mathcal{J}}_{\left|t\right|-1}}\tilde{\mathbb{E}}\left(u_s\right){\parallel }_2}.$$

(19)

The relative expectation also serves as an indicator of the importance of the ANOVA terms. For each ANOVA term in ${\mathcal{J}}_i$, if its relative expectation exceeds a given threshold $to{l}_{anova}$, it is incorporated into the set ${\tilde{\mathcal{J}}}_i$

$${\tilde{\mathcal{J}}}_i:=\left\{t\right|t\in {\mathcal{J}}_i\mathrm{and}{\tilde{\gamma }}_t\ge to{l}_{anova}\}.$$

(20)

Then, the set of active ANOVA indices at order $(i+1)$ can be constructed through

$${\mathcal{J}}_{i+1}:=\left\{t\right|\left|t\right|=i+1\mathrm{and}\forall s\subset t\mathrm{with}\left|s\right|=i\mathrm{satisfies}s\in {\tilde{\mathcal{J}}}_i\}.$$

(21)

This implies that for each selected ANOVA term in $\mathcal{J}$, both the term itself and any of its subsets exhibit relative expectation values greater than the specified threshold. The process automatically concludes when the set of active terms at some order becomes empty. It has been demonstrated that even with a very small threshold, only a small fraction of all expansion terms in $\mathcal{J}$ are reserved, with most of them being low-order terms [13,28].

By performing ANOVA decomposition on $\xi$ of $G\left(\xi \right)$ in Equation (5) using the aforementioned approach, one can decompose the original problem into several low-dimensional sub-problems. Gaussian process regression is then employed for local modeling. Subsequently, Equation (6) is utilized to combine solutions from each sub-problem into the full solution. In the following section, we will delve into the details of Gaussian process regression.

3.2. Gaussian Process Regression

A Gaussian process is a collection of random variables defined over a continuous domain for which any finite combination of these random variables collectively follows a joint Gaussian distribution. It can be conceptualized as an extension of the multidimensional Gaussian distribution to infinite dimensions [29]. Following the representation in [30], a Gaussian process $f\left(x\right)$$(x\in \mathcal{X})$ can be specified by a mean function $m\left(x\right)$ and a covariance function $k(x,x^{{}^{\prime }})$: denoted as $f\left(x\right)\sim \mathcal{GP}(m\left(x\right),k(x,x^{{}^{\prime }}))$. Here, x can be either one-dimensional or multidimensional. Given a series of observation points ${\left\{x_i\right\}}_{i=1}^n\subset \mathcal{X}$, according to the definition of a Gaussian process, ${\left[f\left(x_1\right),\cdots ,f\left(x_n\right)\right]}^{\mathrm{T}}$ is expected to adhere to a multidimensional Gaussian distribution:

$${\left[f\left(x_1\right),\cdots ,f\left(x_n\right)\right]}^{\mathrm{T}}\sim \mathcal{N}(m,K),$$

(22)

where $m={[m\left(x_1\right),\cdots ,m\left(x_n\right)]}^{\mathrm{T}}$ and $K_{ij}=k(x_i,x_j)$. Next, we will delve into the details of using Gaussian processes for regression.

Given a set of observed points $X={[x_1,\cdots ,x_n]}^{\mathrm{T}}$ and their corresponding observed values $y={[y_1,\cdots ,y_n]}^{\mathrm{T}}$ as well as predicted points $Z={[z_1,\cdots ,z_q]}^{\mathrm{T}}$, one can utilize the properties discussed earlier to derive the joint distribution of observed values y and predicted values $f^{*}=f\left(Z\right)={[f\left(z_1\right),\cdots ,f\left(z_q\right)]}^{\mathrm{T}}$:

$$\left[\begin{array}{c}y\hfill \\ f^{*}\hfill \end{array}\right]\sim \mathcal{N}\left(\begin{array}{c}\left[\begin{array}{c}m\left(X\right)\hfill \\ m\left(Z\right)\hfill \end{array}\right],\left[\begin{array}{c}K(X,X)K(X,Z)\hfill \\ K{(X,Z)}^{\mathrm{T}}K(Z,Z)\hfill \end{array}\right]\hfill \end{array}\right),$$

(23)

where $m\left(X\right)={[m\left(x_1\right),\cdots m\left(x_n\right)]}^{\mathrm{T}}$; $m\left(Z\right)={[m\left(z_1\right),\cdots m\left(z_q\right)]}^{\mathrm{T}}$; and the $ij$-th elements of $K(X,Z)$, $K(X,X)$, and $K(Z,Z)$ are $k(x_i,z_j)$, $k(x_i,x_j)$, and $k(z_i,z_j)$, respectively. Thus, the conditional probability distribution of $f^{*}$ is expressed as

$$p\left(f^{*}\right|X,y,Z)\sim \mathcal{N}(\widehat{m},\widehat{\sum }),$$

(24)

where the predicted mean $\widehat{m}$ and the predicted covariance matrix $\widehat{\sum }$ are

$$\begin{array}{cc}\hfill \widehat{m}& =K{(X,Z)}^{\mathrm{T}}{\left(K(X,X)\right)}^{-1}(y-m\left(X\right))+m\left(Z\right),\hfill \\ \hfill \widehat{\sum }& =K(Z,Z)-K{(X,Z)}^{\mathrm{T}}{\left(K(X,X)\right)}^{-1}K(X,Z).\hfill \end{array}$$

(25)

In practice, data are often standardized to be zero mean. Given that $m\left(x\right)=0$, the expressions for the predicted mean and predicted covariance matrix become

$$\begin{array}{cc}\hfill \widehat{m}& =K{(X,Z)}^{\mathrm{T}}{\left(K(X,X)\right)}^{-1}y,\hfill \\ \hfill \widehat{\sum }& =K(Z,Z)-K{(X,Z)}^{\mathrm{T}}{\left(K(X,X)\right)}^{-1}K(X,Z).\hfill \end{array}$$

(26)

The covariance function, commonly referred to as the kernel of the Gaussian process, is evidently crucial in the outlined process. However, determining the specific form and parameters of the kernel poses a challenge. Currently, there is no unified solution for kernel selection across different problems. Researchers typically rely on the experience of their predecessors to manually choose a relatively suitable form. One of the most frequently used kernels in Gaussian processes is the squared exponential kernel, which is also known as the Gaussian kernel [31]:

$$k_{SE}(x,x{}^{\prime })={\rho }^{2}exp\left(-\frac{\parallel x-x{}^{\prime }{\parallel }^2}{2l^2}\right),$$

(27)

where l represents the correlation length, and ${\rho }^2$ is the variance. When extended to high-dimensional situations, where fluctuations in each dimension are not uniform, one can consider an automated relevance determination form for the kernel:

$$k_{SEard}(x,x^{\prime })={\rho }^2\exp \left(-\frac{{(x-x{}^{\prime })}^{\mathrm{T}}{L}^{-1}(x-x{}^{\prime })}{2}\right),$$

(28)

where $x\in {\mathbb{R}}^M$, and $L=\mathrm{diag}(l_1^2,\cdots ,l_M^2)$ is a diagonal matrix with the squared correlation length of each dimension on the diagonal.

With the specified kernel form, the hyperparameters still need to be determined through optimization. Let $\beta ={\left[l_1,\cdots ,l_M,\rho \right]}^{\mathrm{T}}$, and our objective is to maximize the marginal probability of observed values and, equivalently, to minimize its negative logarithm [32]:

$$M\left(\beta \right)=-logp\left(y\right|\beta )=\frac{1}{2}\log \det \left(K\left(\beta \right)\right)+\frac{1}{2}{y}^{\mathrm{T}}{K}^{-1}\left(\beta \right)y+\frac{n}{2}\log \left(2\pi \right),$$

(29)

where $K\left(\beta \right)=K(X,X;\beta )$ is the covariance matrix for a given parameter $\beta$.

As discussed earlier, our study involves Bayesian inversion, necessitating the repeated solution of a partial differential equation with random parameters. To streamline this process, we intend to utilize the finite element method to pre-calculate solutions for a subset of parameters. These parameters and their corresponding finite element solutions will then serve as training samples in the Gaussian process to optimize kernel parameters and establish the Gaussian process model. This approach enables us to predict results directly when confronted with unknown parameters: eliminating the need to solve the equation anew.

However, classical Gaussian process regression faces challenges when handling multi-dimensional outputs. Modeling each dimension separately would imply assuming that each dimension is unrelated [33]. Hence, directly using the finite element solutions as the predicted output in Gaussian process regression may not be advisable. To address this, principal component analysis becomes necessary. It allows us to project the solution space of the equation into a lower-dimensional subspace, thereby improving the performance of Gaussian process regression.

3.3. Principal Component Analysis

This section provides a detailed description of how to employ principal component analysis (PCA) on each sub-problem to reduce the dimensionality of the output of the forward model.

Following the representation in [33], for every ANOVA term $u_t\left({\xi }_t\right)$, assuming that we have $N_{model}$ training samples: $\left\{y_t^{\left(j\right)}\right|y_t^{\left(j\right)}=u_t\left({\xi }_t^{\left(j\right)}\right)\in {\mathbb{R}}^d,j=1,\cdots ,N_{model}\}$. First, standardize these samples to zero mean:

$${\tilde{y}}_t^{\left(j\right)}=y_t^{\left(j\right)}-{\mu }_t,j=1,\cdots ,N_{model},$$

(30)

where

$${\mu }_t=\frac{1}{N_{model}}\sum _{j=1}^{N_{model}}{y}_t^{\left(j\right)}.$$

(31)

Construct the matrix

$$S:=\left[{\tilde{y}}_t^{\left(1\right)},{\tilde{y}}_t^{\left(2\right)},\cdots ,{\tilde{y}}_t^{\left(N_{model}\right)}\right],$$

(32)

and perform singular value decomposition:

$$S=U\sum V^{\mathrm{T}},$$

(33)

where $\sum$ is a diagonal matrix composed of the singular values

$$\sum =\mathrm{diag}\left([{\sigma }_1,{\sigma }_2,\cdots ,{\sigma }_{N_{model}}]\right),{\sigma }_1\ge {\sigma }_2\ge \cdots \ge {\sigma }_{N_{model}}.$$

(34)

Given the threshold $to{l}_{pca}$, one can find $0\le R\le N_{model}$ such that

$$\frac{{\sum }_{j=1}^R{\sigma }_j^2}{{\sum }_{j=1}^{N_{model}}{\sigma }_j^2}>1-to{l}_{pca}\mathrm{and}\frac{{\sum }_{j=1}^{R-1}{\sigma }_j^2}{{\sum }_{j=1}^{N_{model}}{\sigma }_j^2}\le 1-to{l}_{pca}.$$

(35)

Select the first R column vectors of the matrix U as the principal components matrix:

$$V_t:=[v_1,\cdots ,v_R].$$

(36)

 Consequently, $u_t\left({\xi }_t\right)$ can be approximated as

$$u_t\left({\xi }_t\right)\approx V_t{\tilde{u}}_t\left({\xi }_t\right)+{\mu }_t,$$

(37)

where

$$\begin{array}{cc}\hfill {\tilde{u}}_t\left({\xi }_t\right)& ={[{\tilde{u}}_{t,1}\left({\xi }_t\right),\cdots ,{\tilde{u}}_{t,R}\left({\xi }_t\right)]}^{\mathrm{T}}\in {\mathbb{R}}^R,\hfill \\ \hfill {\tilde{u}}_{t,r}\left({\xi }_t\right)& =v_r^{\mathrm{T}}(u_t\left({\xi }_t\right)-{\mu }_t),\mathrm{for}r=1,\cdots ,R.\hfill \end{array}$$

(38)

Generally, singular values decay quite rapidly, making the value of R much smaller than the dimension of the model output. We then construct a Gaussian process model for each PCA mode ${\tilde{u}}_{t,r}\left({\xi }_t\right)(r=1,\cdots ,R)$ and ultimately obtain $u_t\left({\xi }_t\right)$ through Equation (37).

3.4. ANOVA-GP Modeling

This section outlines the overall ANOVA-GP modeling approach [33] used to construct a low-complexity surrogate model for the forward problem (5).

Given the input samples $\Xi :={\left\{{\xi }^{\left(j\right)}\right\}}_{j=1}^{N_{model}}$ generated from a specific distribution, the first step is to conduct the truncated ANOVA expansion as described in Section 3.1. This decomposes the problem with high-dimensional inputs into several low-order sub-problems: yielding the active set $\mathcal{J}$ and training sets ${\{{\xi }_t^{\left(j\right)},u_t\left({\xi }_t^{\left(j\right)}\right)\}}_{j=1}^{N_{model}}$ for $t\in \mathcal{J}$. The second step is to perform PCA analysis as presented in Section 3.3: projecting the high-dimensional outputs $u_t\left({\xi }_t^{\left(j\right)}\right)$ of each sub-problem into a compact subspace denoted as ${\tilde{u}}_t\left({\xi }_t^{\left(j\right)}\right)={[{\tilde{u}}_{t,1}\left({\xi }_t^{\left(j\right)}\right),\cdots ,{\tilde{u}}_{t,R}\left({\xi }_t^{\left(j\right)}\right)]}^{\mathrm{T}}$. The final step is to perform Gaussian process regression for each local training set ${\{{\xi }_t^{\left(j\right)},{\tilde{u}}_{t,r}\left({\xi }_t^{\left(j\right)}\right)\}}_{j=1}^{N_{model}}$, resulting in GP models

$${\widehat{u}}_{t,r}=\mathcal{GP}(m_{t,r}\left({\xi }_t\right),k_{t,r}({\xi }_t,{\xi }_t^{\prime }))$$

(39)

for $t\in \mathcal{J}$ and $r=1,\cdots ,R$. The squared exponential kernel with automated relevance determination is selected as the covariance function here, and the hyperparameter ${\beta }_{t,r}$ is determined through minimizing

$$\begin{array}{cc}\hfill M\left({\beta }_{t,r}\right)& =-logp\left({\alpha }_{t,r}\right|{\beta }_{t,r})\hfill \\ \hfill & =\frac{1}{2}\log \det \left(K\left({\beta }_{t,r}\right)\right)+\frac{1}{2}{\alpha }_{t,r}^{\mathrm{T}}{K}^{-1}\left({\beta }_{t,r}\right){\alpha }_{t,r}+\frac{N_{model}}{2}\log \left(2\pi \right),\hfill \end{array}$$

(40)

where

$${\alpha }_{t,r}={[{\tilde{u}}_{t,r}\left({\xi }_t^{\left(1\right)}\right),\cdots ,{\tilde{u}}_{t,r}\left({\xi }_t^{\left(N_{model}\right)}\right)]}^{\mathrm{T}},K{\left({\beta }_{t,r}\right)}_{ij}=k_{t,r}({\xi }_t^{\left(i\right)},{\xi }_t^{\left(j\right)}).$$

(41)

By substituting the predicted mean values of these models into Equation (37), we obtain the predictor of $u_t\left({\xi }_t\right)$:

$${\widehat{u}}_t\left({\xi }_t\right)=V_t{U}_t\left({\xi }_t\right)+{\mu }_t,U_t\left({\xi }_t\right)={[{\widehat{m}}_{t,1}\left({\xi }_t\right),\cdots ,{\widehat{m}}_{t,R}\left({\xi }_t\right)]}^{\mathrm{T}},$$

(42)

where

$${\widehat{m}}_{t,r}\left({\xi }_t\right)=k_{*}^{\mathrm{T}}{K}^{-1}\left({\beta }_{t,r}\right){\alpha }_{t,r},k_{*}={[k_{t,r}({\xi }_t,{\xi }_t^{\left(1\right)}),\cdots ,k_{t,r}({\xi }_t,{\xi }_t^{\left(N_{model}\right)})]}^{\mathrm{T}}.$$

(43)

 Finally, in accordance with Equation (14), we integrate all local predictors of sub-problems to obtain the overall ANOVA-GP model:

$${\widehat{u}}_{\mathcal{J}}\left(\xi \right)=u_0+\sum _{t\in \mathcal{J}}{\widehat{u}}_t\left({\xi }_t\right).$$

(44)

Refer to Algorithm 2 for a detailed description of the entire process of obtaining the ANOVA-GP model.

Algorithm 2 Construction of ANOVA-GP model

Input: Sample set $\Xi ={\left\{{\xi }^{\left(j\right)}\right\}}_{i=1}^{N_{model}}\subset I^M$. Output: ANOVA-GP model ${\widehat{u}}_{\mathcal{J}}$. 1: Compute $c=\frac{1}{N_{model}}{\sum }_{j=1}^{N_{model}}{\xi }^{\left(j\right)}$. 2: Compute the zero-order term $u_0=u\left(c\right)$. 3: Initialize $\mathcal{J}={\mathcal{J}}_1=\{1,\cdots ,M\}$ and $i=1$. 4: while  ${\mathcal{J}}_i\ne \varnothing$  do 5:     Let ${\tilde{\mathcal{J}}}_i=\varnothing$. 6:     for $t\in {\mathcal{J}}_i$ do 7:         Construct sample set ${\Xi }_t=\{{\xi }_t^{\left(j\right)},{\xi }^{\left(j\right)}\in \Xi \mathrm{for}j=1,\cdots ,N_{model}\}\subset I^i$. 8:         Compute $\widetilde{{\gamma }_t}=\frac{\parallel \tilde{\mathbb{E}}\left(u_t\right)\parallel }{\parallel u_0+{\sum }_{s\in {\mathcal{J}}_1\cup \cdots \cup {\mathcal{J}}_{\left|t\right|-1}}\tilde{\mathbb{E}}\left(u_s\right)\parallel }$ through Equations (8) and (18). 9:         if ${\tilde{\gamma }}_t>to{l}_{anova}$ then 10:            Update ${\tilde{\mathcal{J}}}_i={\tilde{\mathcal{J}}}_i\cup \left\{t\right\}$. 11:         end if 12:     end for 13:     Construct ${\mathcal{J}}_{i+1}=\left\{t\right|\left|t\right|=i+1\mathrm{and}\forall s\subset t\mathrm{with}\left|s\right|=i\mathrm{satisfies}s\in {\tilde{\mathcal{J}}}_i\}$. 14:     Update $\mathcal{J}=\mathcal{J}\cup {\mathcal{J}}_{i+1}$ and $i=i+1$. 15: end while 16: for  $t\in \mathcal{J}$  do 17:     For sample set ${\left\{u_t\left({\xi }_t^{\left(j\right)}\right)\right\}}_{j=1}^{N_{model}}$, obtain the mean ${\mu }_t$, principal components matrix $V_t$, and PCA coefficients ${\left\{{\tilde{u}}_{t,r}\left({\xi }_t^{\left(j\right)}\right)\right\}}_{j=1}^{N_{model}}$ through Equations (30)–(38). 18:     for $r=1:R$ do 19:         Train a Gaussian process regression model ${\widehat{u}}_{t,r}=\mathcal{GP}(m_{t,r}\left({\xi }_t\right),k_{t,r}({\xi }_t,{\xi }_t^{\prime }))$ with training set ${\{{\xi }_t^{\left(j\right)},{\tilde{u}}_{t,r}^{\left(j\right)}\}}_{j=1}^{N_{model}}$, for which the hyperparameter is determined by minimizing (40). 20:     end for 21:     Construct the local predictor ${\widehat{u}}_t\left({\xi }_t\right)$ through Equations (41)–(43). 22: end for 23: Integrate the overall ANOVA-GP model ${\widehat{u}}_{\mathcal{J}}\left(\xi \right)=u_0+{\sum }_{t\in \mathcal{J}}{\widehat{u}}_t\left({\xi }_t\right)$.

3.5. Adaptive ANOVA-GP-MCMC

Section 3.4 introduces a surrogate modeling strategy with the aim of replacing the computationally intensive forward model with fast ANOVA-GP predictions in MCMC iterations. This is done to reduce computational costs and enhance sampling efficiency. Ideally, the sample set $\Xi$ should be selected from the posterior distribution. However, this is impractical since we lack knowledge of the posterior information of the parameters before conducting MCMC. Thus, prior to obtaining posterior information, an initial model must be built using samples that align with the prior probability distribution. Nevertheless, using a model based solely on priors may be inefficient, especially when the prior distribution significantly differs from the posterior distribution [34]. The impact of employing a sample set that does not conform to the actual distribution on model efficiency is primarily observed in two aspects. Firstly, in the anchored ANOVA decomposition, the selection of the anchor point and active terms involves statistical information about parameters. Research by Gal et al. [15] indicates that appropriate anchor point selection—specifically, choosing the expectation of the parameters (i.e., the posterior mean in the Bayesian inverse problem setting)—enables achieving a given approximation accuracy with very few expansion terms. Additionally, the selection of active terms relies on their relative expectations. Therefore, utilizing a sample set that adheres to the posterior distribution leads to a more efficient ANOVA representation. Secondly, during the Gaussian process regression, if the training sample set deviates significantly from the actual distribution of parameters, it affects the accuracy of prediction. To address this issue, we design an adaptive algorithm that updates our ANOVA-GP model based on the obtained posterior samples during the MCMC process to enhance computational efficiency. We refer to this approach as the adaptive ANOVA-GP-MCMC algorithm.

This process is detailed in Algorithm 3. Initially, we obtain $N_{model}$ samples from the prior distribution $\pi \left(\xi \right)$ and construct an initial ANOVA-GP model based on these samples. Then we employ the resulting model as the initial surrogate of the forward model in MCMC.

Algorithm 3 Adaptive ANOVA-GP-MCMC algorithm

Input: Number of MCMC samples N, number of samples to construct model $N_{model}$, number of samples for updating model $N_{update}$, noise distribution ${\pi }_{ϵ}$, proposal distribution ${\pi }_p$, prior distribution $\pi$, and observation d. Output: Posterior samples ${\left\{{\xi }^{\left(j\right)}\right\}}_{j=1}^N$. 1: Generate $N_{model}$ samples from $\pi \left(\xi \right)$ and construct the prior ANOVA-GP model ${\widehat{u}}_{\mathcal{J}}$ according to Algorithm 2. 2: Draw a sample ${\xi }^{\left(0\right)}$ from the prior distribution $\pi$ as the initial state. 3: Let $Updat{e}_{-}label=1$. 4: for  $j=1:N$  do 5:     Generate the proposal ${\xi }^{*}\sim {\pi }_p(·|{\xi }^{(j-1)})$. 6:     Compute the acceptance rate $a:=min\left(1,\frac{{\pi }_{ϵ}(d-{\widehat{u}}_{\mathcal{J}}\left({\xi }^{*}\right))\pi \left({\xi }^{*}\right)}{{\pi }_{ϵ}(d-{\widehat{u}}_{\mathcal{J}}\left({\xi }^{(j-1)}\right))\pi \left({\xi }^{(j-1)}\right)}·\frac{{\pi }_p\left({\xi }^{(j-1)}\right|{\xi }^{*})}{{\pi }_p\left({\xi }^{*}\right|{\xi }^{(j-1)})}\right)$. 7:     Draw $\rho \sim \mathrm{Uniform}[0,1]$. 8:     if $\rho <a$ then 9:         Let ${\xi }^{\left(j\right)}={\xi }^{*}$. 10:     else 11:         Let ${\xi }^{\left(j\right)}={\xi }^{(j-1)}$. 12:     end if 13:     if $j\mathrm{mod}{N}_{update}=0\mathrm{and}Updat{e}_{-}label=1$ then 14:         Let ${\mathcal{J}}^{{}^{\prime }}=\mathcal{J}$. 15:         Randomly select $N_{model}$ samples from ${\left\{{\xi }^{\left(k\right)}\right\}}_{k=j-N_{update}+1}^j$, reconstruct the ANOVA-GP model ${\widehat{u}}_{\mathcal{J}}$, and update $\mathcal{J}$. 16:         if $\mathcal{J}={\mathcal{J}}^{{}^{\prime }}$ then 17:            Let $Updat{e}_{-}label=0$. 18:         end if 19:     end if 20: end for

Suppose we have observation d. Initialize the Markov chain ${\Xi }^{*}:=\left\{{\xi }^{\left(1\right)}\right\}$, where ${\xi }^{\left(1\right)}$ is obtained from $\pi \left(\xi \right)$. Subsequently, for $j\ge 1$, obtain a candidate sample ${\xi }^{*}$ from the proposal distribution ${\pi }_p(·|{\xi }^{\left(j\right)})$ based on the current sample ${\xi }^{\left(j\right)}$. Then, adopt the prediction ${\widehat{u}}_{\mathcal{J}}\left(\xi \right)$ of the ANOVA-GP model to approximate the output $u\left(\xi \right)$ of the forward model. Compute the acceptance rate of the candidate sample ${\xi }^{*}$:

$$a:=min\left(1,\frac{{\pi }_{ϵ}(d-{\widehat{u}}_{\mathcal{J}}\left({\xi }^{*}\right))\pi \left({\xi }^{*}\right)}{{\pi }_{ϵ}(d-{\widehat{u}}_{\mathcal{J}}\left({\xi }^{\left(j\right)}\right))\pi \left({\xi }^{\left(j\right)}\right)}·\frac{{\pi }_p\left({\xi }^{\left(j\right)}\right|{\xi }^{*})}{{\pi }_p\left({\xi }^{*}\right|{\xi }^{\left(j\right)})}\right),$$

(45)

which is the probability that the candidate sample ${\xi }^{*}$ is accepted. If accepted, the state of the Markov chain transits to ${\xi }^{*}$, i.e., ${\xi }^{(j+1)}={\xi }^{*}$; otherwise, it remains the same, i.e., ${\xi }^{(j+1)}={\xi }^{\left(j\right)}$. Each time $N_{update}$ new samples are generated, we randomly select $N_{model}$ samples from them to reconstruct the ANOVA-GP model. This updating procedure stops when the active set $\mathcal{J}$ of the newly updated model remains unchanged.

4. Numerical Study

In this section, we consider a Bayesian inverse problem in which the forward model is governed by partial differential equations with random parameters. To demonstrate the efficiency and effectiveness of the ANOVA-GP modeling approach and the proposed adaptive ANOVA-GP-MCMC algorithm, both the forward and the inverse problems are investigated. In this section, the finite element solutions are obtained by IFISS [35,36]. All the results presented here are obtained on a desktop with a 2.20 GHz Intel Xeon E52630 v4 CPU.

4.1. Problem Setup

Consider a parametric two-dimensional elliptic problem on a unit square $D=(0,1)\times (0,1)$:

$$\nabla ·(-exp\left(a(x,\xi )\right)\nabla s(x,\xi ))=f\left(x\right),x\in D,\xi \in {\mathbb{R}}^M$$

(46)

with boundary conditions

$$\left\{\begin{array}{cc}\hfill exp\left(a(x,\xi )\right)\nabla s(x,\xi )·\overrightarrow{n}=0,& x\in (0,1)\times \{0,1\}\hfill \\ \hfill s(x,\xi )=0,& x\in \{0,1\}\times [0,1],\hfill \end{array}\right.$$

(47)

where $\overrightarrow{n}$ denotes the unit normal vector. In (47), the notation $(0,1)$ represents the open interval of real numbers from 0 to 1, while $[0,1]$ denotes the closed interval. The set $\{0,1\}$ consists of the real numbers 0 and 1. This equation characterizes steady flows in porous media, where $a(x,\xi )$ is an unknown permeability field, $s(x,\xi )$ represents the pressure head, and $f\left(x\right)$ serves as the source term, which we set to

$$f\left(x\right)=3exp\left(-{(x_1-0.5)}^2-{(x_2-0.5)}^2\right).$$

(48)

The permeability field $a(x,\xi )$ is parameterized through a truncated Karhunen–Loève (KL) expansion [18] of a Gaussian random field featuring a constant mean function $a_0\left(x\right)$ and an exponential kernel covariance function.

$$C(x,x^{\prime })={\sigma }_f^{2}exp\left(-\frac{\parallel x-x^{\prime }{\parallel }_2^2}{\tau }\right),$$

(49)

where ${\sigma }_f$ represents the standard deviation and $\tau$ is the correlation length. Denoting the eigenpairs of the covariance function (49) as ${\{{\lambda }_k,a_k\left(x\right)\}}_{k=1}^{\infty }$, the truncated Karhunen–Loève expansion can be expressed as

$$a(x,\xi )=a_0\left(x\right)+\sum _{k=1}^M\sqrt{{\lambda }_k}{a}_k\left(x\right){\xi }_k,$$

(50)

where ${\left\{{\xi }_k\right\}}_{k=1}^M$ are independent and identically distributed (i.i.d.) random variables following a standard normal distribution $\mathcal{N}(0,1)$. In this numerical example, we set $a_0\left(x\right)=1$, ${\sigma }_f^2=0.5$, and $\tau =2$. To capture $95\%$ of the total variance of the random field, the KL expansion is truncated at $M=12$. The prior distribution of the parameters ${\left\{{\xi }_k\right\}}_{k=1}^M$ is specified as $\mathcal{N}(0,1)$.

Once $\xi$ is specified, we employ the finite element method [35,36] with bilinear basis functions to numerically solve this equation. The forward model is defined as

$$G\left(\xi \right):=u\left(\xi \right)=\mathfrak{c}\left(s\left(\xi \right)\right)=\{s(x,\xi ),x\in S\}\in {\mathbb{R}}^d,$$

(51)

where S is the collection of sensor points in the physical space and is defined as $S:=\{(x_1,x_2)\in D\mid x_1=0.125i,x_2=0.125j,\forall i,j=1,2,\cdots ,7\}$. Consequently, the dimension of the model output is 49. To generate the ground truth of the permeability field $a^{*}\left(x\right)$ utilized in the inverse problem, we randomly generate ${\left\{{\xi }_k^{*}\right\}}_{k=1}^M$ from the prior distribution $\mathcal{N}(0,1)$ and assemble them through (50). Figure 1 illustrates the discretization of the physical domain and the locations of sensors as well as the actual permeability field $a^{*}\left(x\right)$.

Figure 1. The FEM grids and sensor locations (left) and the ground truth of the permeability field (right).

4.2. Performance in Forward Problem

This section assesses the accuracy and efficiency of the ANOVA-GP models. As outlined in Section 3.4, the construction of an ANOVA-GP model necessitates samples of the parameters. In the absence of posterior information, the prior distribution of the parameters is considered. The anchor point c is chosen as the prior mean of the parameters, and the sample set ${\left\{{\xi }^{\left(j\right)}\right\}}_{j=1}^{N_{model}}$ consists of $N_{model}$ samples randomly drawn from the prior distribution. The number of samples is set to $N_{model}=100$. The threshold for retaining PCA modes is defined as $to{l}_{pca}={10}^{-3}$. We construct three ANOVA-GP models with different thresholds $to{l}_{anova}={10}^{-2},{10}^{-3},{10}^{-4}$ for selecting active ANOVA terms and employ the following methods to evaluate their performance.

The test samples ${\left\{{\xi }^{\left(j\right)}\right\}}_{j=1}^{N_{test}}$ are randomly drawn from the prior distribution, with $N_{test}=500$. The reference solutions $u\left({\xi }^{\left(j\right)}\right)$ are obtained using the finite element method. The relative error between the predicted mean ${\overline{u}}_{\mathcal{J}}\left({\xi }^{\left(j\right)}\right)$ given by the ANOVA-GP model and the reference solution is defined as

$$\mathrm{Relative}\mathrm{error}=\frac{\parallel u\left({\xi }^{\left(j\right)}\right)-{\overline{u}}_{\mathcal{J}}\left({\xi }^{\left(j\right)}\right){\parallel }_2}{\parallel u\left({\xi }^{\left(j\right)}\right){\parallel }_2}.$$

(52)

Table 1 and Figure 2 demonstrate that as $to{l}_{anova}$ decreases, more ANOVA terms are included in $\mathcal{J}$: leading to improved accuracy but at the expense of increased prediction cost. To strike a balance between accuracy and efficiency, $to{l}_{anova}={10}^{-3}$ is adopted in subsequent experiments.

Table 1. The mean of relative errors and the mean of prediction costs of ANOVA-GP models when $to{l}_{anova}={10}^{-2}$ (left), ${10}^{-3}$ (middle), and ${10}^{-4}$ (right).

	${\mathit{tol}}_{\mathit{anova}}={10}^{-2}$	${\mathit{tol}}_{\mathit{anova}}={10}^{-3}$	${\mathit{tol}}_{\mathit{anova}}={10}^{-4}$
Mean cost (s)	$1.279\times {10}^{-2}$	$1.684\times {10}^{-2}$	$2.184\times {10}^{-2}$
Mean relative error	$6.531\times {10}^{-3}$	$3.992\times {10}^{-3}$	$1.114\times {10}^{-3}$

Figure 2. The relative errors of ANOVA-GP models when $to{l}_{anova}={10}^{-2}$ (left), ${10}^{-3}$ (middle), and ${10}^{-4}$ (right) for 500 test samples.

As mentioned in Section 3.1.2, $\mathcal{J}$ comprises only a few low-order terms. Table 2 provides the counts of active terms in ${\mathcal{J}}_i$ and $\widetilde{{\mathcal{J}}_i}$ at each order $(i=1,2,3)$ as well as for the overall set $\mathcal{J}$. In all settings, only terms up to second-order are selected into $\mathcal{J}$, which is consistent with findings from prior research [13].

Table 2. The number of active ANOVA terms at each order when $to{l}_{anova}={10}^{-2}$ (left), ${10}^{-3}$ (middle), and ${10}^{-4}$ (right).

${\mathit{tol}}_{\mathit{anova}}$	$|{\mathcal{J}}_1|$	$|{\tilde{\mathcal{J}}}_1|$	$|{\mathcal{J}}_2|$	$|{\tilde{\mathcal{J}}}_2|$	$|{\mathcal{J}}_3|$	$\left|\mathcal{J}\right|$
${\mathbf{10}}^{-\mathbf{2}}$	12	1	0	0	0	12
${\mathbf{10}}^{-\mathbf{3}}$	12	3	3	0	0	15
${\mathbf{10}}^{-\mathbf{4}}$	12	4	6	3	0	18

4.3. Performance in Inverse Problem

This section assesses the performance of the ANOVA-GP model in MCMC and verifies the validity of the adaptive ANOVA-GP-MCMC algorithm (see Algorithm 3).

Three sets of MCMC inversions are carried out in the numerical study. The first set utilizes the finite element method (referred to as the full model in the subsequent text) in the forward evaluation. The second set employs the ANOVA-GP model constructed with prior samples (refer to Section 4.2) as a surrogate for the forward model. The third set adopts the adaptive ANOVA-GP-MCMC algorithm (see Algorithm 3), where the ANOVA-GP model is adaptively updated during sampling. These three sets are denoted as full MCMC, prior ANOVA-GP-MCMC, and adaptive ANOVA-GP-MCMC, respectively, in the subsequent text.

In all cases, ${10}^6$ samples are generated through MCMC. The proposal distribution ${\pi }_p(·|{\xi }^{\left(j\right)})$ is set to a multivariate Gaussian distribution with a mean value ${\xi }^{\left(j\right)}$ and a covariance matrix of $0.{015}^{2}I$. The measurement noise of sensors is modeled as independent and identically distributed Gaussian distributions $ϵ\sim \mathcal{N}(0,0.{001}^2)$. For constructing the ANOVA-GP surrogate, we set $N_{model}=100$. In adaptive ANOVA-GP-MCMC, the model is updated after $N_{update}={10}^3$ new samples are generated until the set of active terms $\mathcal{J}$ remains unchanged. The acceptance rates for the three groups are $31.13\%$, $31.25\%$, and $30.98\%$, respectively.

Figure 3 displays the mean of posterior samples generated by full MCMC, prior ANOVA-GP-MCMC, and adaptive ANOVA-GP-MCMC, from which we can observe that the results obtained from the three sets are similar to the ground truth (see Figure 1), indicating the effectiveness of the proposed approach.

Figure 3. The means of posterior samples generated by full MCMC (left), prior ANOVA-GP-MCMC (middle), and adaptive ANOVA-GP-MCMC (right).

Table 3 illustrates the average cost for a single sample in the three sets. The model based on posterior samples accelerates the sampling speed by $82.73\%$, evidencing the substantial improvement in efficiency achieved by the proposed method.

Table 3. The mean costs per sample of full MCMC (left), prior ANOVA-GP-MCMC (middle), and adaptive ANOVA-GP-MCMC (right).

	Full	Prior	Adaptive
Cost per sample (s)	$4.663\times {10}^{-2}$	$2.414\times {10}^{-2}$	$8.051\times {10}^{-3}$
Speedup	∖	$48.23\%$	$82.73\%$

To access the accuracy of the estimated posterior mean and variance, we introduce the relative mean error ${ϵ}_{\mathrm{mean}}$ and the relative variance error ${ϵ}_{\mathrm{var}}$ as

$${ϵ}_{\mathrm{mean}}:=\frac{\parallel {\mathbf{E}}_{{\Xi }^{*}}\left(a(x,\xi )\right)-{\mathbf{E}}_{\mathrm{ref}}{\parallel }_2}{\parallel {\mathbf{E}}_{\mathrm{ref}}{\parallel }_2},{ϵ}_{\mathrm{var}}:=\frac{\parallel {\mathbf{V}}_{{\Xi }^{*}}\left(a(x,\xi )\right)-{\mathbf{V}}_{\mathrm{ref}}{\parallel }_2}{\parallel {\mathbf{V}}_{\mathrm{ref}}{\parallel }_2},$$

(53)

where the mean and variance of the posterior samples are computed through

$$\begin{array}{cc}\hfill {\mathbf{E}}_{{\Xi }^{*}}\left(a(x,\xi )\right)& :=\frac{1}{|{\Xi }^{*}|}\sum _{j=1}^{|{\Xi }^{*}|}a(x,{\xi }^{\left(j\right)}),\hfill \\ \hfill {\mathbf{V}}_{{\Xi }^{*}}\left(a(x,\xi )\right)& :=\frac{1}{\left|{\Xi }^{*}\right|}\sum _{j=1}^{\left|{\Xi }^{*}\right|}{(a(x,{\xi }^{\left(j\right)})-{\mathbf{E}}_{{\Xi }^{*}}\left(a(x,\xi )\right))}^2.\hfill \end{array}$$

(54)

The reference mean estimate ${\mathbf{E}}_{\mathrm{ref}}$ and the reference variance estimate ${\mathbf{V}}_{\mathrm{ref}}$ in (53) are obtained by full MCMC with ${10}^6$ samples.

Figure 4 illustrates the trend of ${ϵ}_{\mathrm{mean}}$ and ${ϵ}_{\mathrm{var}}$ during the sampling process of the three sets. The figure reveals that, owing to the acceleration of individual sampling, the relative error decreases more rapidly using the ANOVA-GP model than with the full model. Particularly in the adaptive ANOVA-GP model, as the anchor point and the samples used for construction are closer to the posterior distribution, fewer ANOVA terms are selected in $\mathcal{J}$, and a smaller number of PCA modes remain in each local problem.

Figure 4. Errors in mean and variance estimates (${ϵ}_{\mathrm{mean}}$ and ${ϵ}_{\mathrm{var}}$) for full MCMC, prior ANOVA-GP-MCMC, and adaptive ANOVA-GP-MCMC.

Table 4 shows the total number of PCA modes in the model used in prior ANOVA-GP-MCMC (referred to as the prior ANOVA-GP model) and the updating model in adaptive ANOVA-GP-MCMC (referred to as the posterior ANOVA-GP model). Because of the substantial reduction in the cost of a single prediction compared to the prior-based model, adaptive ANOVA-GP-MCMC achieves a given accuracy even more rapidly.

Table 4. The number of total PCA modes in the prior ANOVA-GP model (left) and the posterior ANOVA-GP model (right).

	Prior ANOVA-GP Model	Posterior ANOVA-GP Model
Number of PCA modes	47	29

5. Conclusions

In high-dimensional Bayesian inverse problems, establishing a cost-effective surrogate model for the forward model is pivotal for accelerating the MCMC process. Our study leverages ANOVA decomposition and PCA to break down the high-dimensional original problem into several low-dimensional input–output sub-problems. Gaussian process regression is then employed to build local predictors. To enhance the overall efficiency of the model in the Bayesian inversion setting, we propose an adaptive ANOVA-GP-MCMC algorithm that updates the model using posterior samples obtained during the sampling process. Through numerical experiments, we first validate the computational accuracy and efficiency of the ANOVA-GP model. Subsequently, we apply this model to the MCMC process to evaluate the effectiveness of our adaptive algorithm. It is important to note that our study currently focuses on ANOVA decomposition based on a single anchor point: limiting its applicability to Bayesian inverse problems with single-peak posterior distributions. For multi-peak distributions, a potential solution is to employ multiple anchor points for ANOVA decomposition. Additionally, although the number of selected ANOVA terms does not increase exponentially with the growth of the parameter dimension, this method may still lose its acceleration effect when dealing with extremely high-dimensional parameters. Developing a high-dimensional-model reduction method with broader applicability than ANOVA decomposition will be a key focus of our future work.