A Stochastic Discrete Empirical Interpolation Approach for Parameterized Systems

Abstract

As efficient separation of variables plays a central role in model reduction for nonlinear and nonaffine parameterized systems, we propose a stochastic discrete empirical interpolation method (SDEIM) for this purpose. In our SDEIM, candidate basis functions are generated through a random sampling procedure, and the dimension of the approximation space is systematically determined by a probability threshold. This random sampling procedure avoids large candidate sample sets for high-dimensional parameters, and the probability based stopping criterion can efficiently control the dimension of the approximation space. Numerical experiments are conducted to demonstrate the computational efficiency of SDEIM, which include separation of variables for general nonlinear functions, e.g., exponential functions of the Karhu nen–Loève (KL) expansion, and constructing reduced order models for FitzHugh–Nagumo equations, where symmetry among limit cycles is well captured by SDEIM.

1. Introduction

When conducting model reduction for nonlinear and nonaffine parameterized systems [1], separation of variables is an important step. During the last few decades, many strategies have been developed to achieve this goal, e.g., empirical interpolation methods (EIM) [2,3,4,5], discrete empirical interpolation methods (DEIM) [6,7], and variable-separation (VS) methods [8]. To result in an accurate linear representation, these methods typically need a fine reduced basis approximation space. To construct the corresponding basis functions, repeated evaluations of expensive parameterized systems are required. The accuracy of the EIM/DEIM approximations depends on candidate parameter samples. Properly choosing the samples is crucial and is especially challenging when the parameter space is high dimensional.

For reduced basis approximations [9,10,11,12,13,14], the work in [15] shows that candidate sample sets for reduced bases can be chosen as a random set of a specified order and the resulting approximation satisfies given accuracies with a probability threshold. Rather than reduced basis approximations, we in this paper focus on separation of variables for nonlinear and nonaffine systems, and propose a stochastic discrete empirical interpolation method (SDEIM). In our SDEIM, the interpolation is processed through two steps: the first is to randomly select sample points to construct an approximation space for empirical interpolation, and the second is to evaluate if the approximation accuracy meets the probability threshold on additional samples. These two steps are repeated until the approximation space satisfies the given accuracy and probability requirements. The probability correctness of the approximation space is reassessed by SDEIM each time the approximation space is updated. We note that SDEIM does not loop over a fine candidate sample set, which significantly reduces the computational cost for empirical interpolation for high-dimensional systems. To demonstrate the efficiency of SDEIM, we utilize it to approximate test functions including the exponential functions of the Karhu nen–Loève (KL) expansion, which is widely used to parameterize random fields. Finally, we use SDEIM to solve ordinary differential equations (ODEs) arising from the FitzHugh–Nagumo (F–N) system [6,16], and all these results show that SDEIM is efficient. It is worth noting that many natural phenomena can have symmetric properties, and symmetry of limit cycles of the F-N system is well captured by SDEIM.

An outline of the paper is as follows. We first present our problem setting, and then review the fine collateral reduced-basis approximation space and EIM in Section 2. After a short introduction of the discrete form, we present SDEIM and analyze its performance in Section 3. Numerical results are discussed in Section 4. Finally, we conclude the paper in Section 5.

2. Problem Formulation

Nonlinear terms and functions in complex systems cause significant difficulties for efficient model reduction, and separation of variables is typically required. In this section, we refer to nonlinear functions under consideration as the target functions. Let $\mathsf{\Omega }\subset {\mathbb{R}}^d$ (for $d=1,2$ or 3) be a bounded, connected physical domain and $\mathsf{\Gamma }$ be a high dimensional parameter space. The target function with a general form is written as

$$\xi \mapsto f(x,\xi ),x\in \mathsf{\Omega },\xi \in \mathsf{\Gamma }.$$

The target function $f(x,\xi )$ is assumed to be nonlinear. In the following, we review EIM [2], which is to approximate $f(x,\xi )$ with a separate representation for x and $\xi$.

2.1. The Linear Approximation Space

Before introducing EIM, we first introduce its fine collateral reduced-basis approximation space following [2,6,17,18]. Without loss of generality, we assume that the target function $f(x,\xi )$ is uniquely defined in some Hilbert space $\mathcal{H}$ for every $\xi \in \mathsf{\Gamma }$. An inner product and a $L_2$ norm of $\mathcal{H}$ are denoted as $<f,g>:={\int }_{\mathsf{\Omega }}f(x,\xi )g(x,\xi )\mathrm{d}x$ and $\parallel ·\parallel :=\sqrt{<·,·>}$. We define the target manifold $\mathcal{M}$ as

$$\mathcal{M}:=\left\{f(x,\xi )|\xi \in \mathsf{\Gamma }\right\}.$$

when the target manifold $\mathcal{M}$ has a low dimension, a low dimensional linear space can approximate it well. The infimum of the supremum distance between the n-dimensional linear approximation space and the target manifold $\mathcal{M}$ is called the Kolmogorov n-width [3,19], which is defined as

$${\kappa }_n={\kappa }_n\left(\mathcal{M}\right):=\underset{dim\left(\mathcal{V}\right)=n}{inf}\underset{\xi \in \mathsf{\Gamma }}{sup}∥f(x,\xi )-{\mathcal{P}}_{\mathcal{V}}f(x,\xi )∥.$$

(1)

Here, ${\mathcal{P}}_{\mathcal{V}}$ denotes the orthogonal projection onto $\mathcal{V}$, where $\mathcal{V}$ is an arbitrary linear space, i.e.,

$$<{\mathcal{P}}_{\mathcal{V}}f(x,\xi ),f(x,\xi )-{\mathcal{P}}_{\mathcal{V}}f(x,\xi )>=0.$$

(2)

Moreover, from (2) we can get the property

$$\begin{array}{cc}\hfill \parallel {\mathcal{P}}_{\mathcal{V}}{f(x,\xi )\parallel }^2& =<{\mathcal{P}}_{\mathcal{V}}f(x,\xi ),{\mathcal{P}}_{\mathcal{V}}f(x,\xi )>\hfill \\ \hfill & =⟨f(x,\xi ),f(x,\xi )⟩-⟨f(x,\xi )-{\mathcal{P}}_{\mathcal{V}}f(x,\xi ),f(x,\xi )-{\mathcal{P}}_{\mathcal{V}}f(x,\xi )⟩\hfill \\ \hfill & ={\parallel f(x,\xi )\parallel }^2-{\parallel f(x,\xi )-{\mathcal{P}}_{\mathcal{V}}f(x,\xi )\parallel }^2\hfill \\ \hfill & \le {\parallel f(x,\xi )\parallel }^2.\hfill \end{array}$$

(3)

Since the orthogonal projection is linear, (3) means that the error is reduced after the orthogonal projection. However, the space satisfying (1) is nontrivial to reach. The work in [20] has proposed a specific greedy procedure for constructing this n dimensional linear approximation space which optimizes

$${\tilde{\kappa }}_n:=\underset{dim\left(\mathcal{V}\right)=n}{inf}\underset{\xi \in {\mathsf{\Gamma }}_{\mathrm{train}}}{sup}∥f(x,\xi )-{\mathcal{P}}_{\mathcal{V}}f(x,\xi )∥$$

(4)

instead of ${\kappa }_n$ in a sufficiently fine candidate sample set ${\mathsf{\Gamma }}_{\mathrm{train}}\subset \mathsf{\Gamma }$. Since the candidate set ${\mathsf{\Gamma }}_{\mathrm{train}}$ contains a finite number of samples, optimizing ${\tilde{\kappa }}_n$ is equivalent to search ${\xi }^{\left(k\right)}$ such that

$${\xi }^{\left(k\right)}={argmax}_{\xi \in {\mathsf{\Gamma }}_{\mathrm{train}}}\parallel f(x,\xi )-{\mathcal{P}}_{\mathcal{V}}f(x,\xi )\parallel .$$

Then, we update the linear approximation space $\mathcal{V}$ for this target function $f\left(x,{\xi }^{\left(k\right)}\right)$. A greedy procedure which constructs a linear approximation space $\mathcal{V}$ satisfying ${\tilde{\kappa }}_n\le \frac{tol}{3}$, where $tol$ is a given tolerance, can be stated as follows. First this procedure initializes the linear approximation space ${\mathcal{V}}_0:=span\left(\left\{0\right\}\right)$. Then, it selects a new parameter value by

$${\xi }^{(k+1)}:={argmax}_{\xi \in {\mathsf{\Gamma }}_{\mathrm{train}}}\parallel f(x,\xi )-{\mathcal{P}}_{{\mathcal{V}}_k}f(x,\xi )\parallel .$$

If the error

$$\tilde{\eta }\left({\xi }^{(k+1)}\right)=∥f\left(x,{\xi }^{(k+1)}\right)-{\mathcal{P}}_{{\mathcal{V}}_k}f\left(x,{\xi }^{(k+1)}\right)∥$$

is greater than $\frac{tol}{3}$, we update the linear approximation space ${\mathcal{V}}_{k+1}=span\left({\mathcal{V}}_k\cup f\left(x,{\xi }^{(k+1)}\right)\right)$. At the end of this procedure, the linear approximation space $\mathcal{V}$ is set to $\mathcal{V}={\mathcal{V}}_n$ and $\mathcal{V}$ satisfies $\tilde{\eta }\left(\xi \right)\le \frac{tol}{3}$ for all $\xi \in {\mathsf{\Gamma }}_{\mathrm{train}}$. Moreover, that the candidate sample set ${\mathsf{\Gamma }}_{\mathrm{train}}$ is sufficiently fine means that the discrete target manifold

$$\tilde{\mathcal{M}}=\left\{f(x,\xi )|\xi \in {\mathsf{\Gamma }}_{\mathrm{train}}\right\}$$

is a $\frac{tol}{3}$-approximation net for target manifold $\mathcal{M}$, i.e.,

$$d(\mathcal{M},\tilde{\mathcal{M}}):=\underset{\xi \in \mathsf{\Gamma }}{sup}\underset{\tilde{\xi }\in {\mathsf{\Gamma }}_{\mathrm{train}}}{inf}∥f(x,\xi )-f(x,\tilde{\xi })∥\le \frac{tol}{3}.$$

(5)

Then for any $\xi \in \mathsf{\Gamma }$, (3) and (5) give

$$\begin{array}{cc}\hfill ∥f(x,\xi )-{\mathcal{P}}_{\mathcal{V}}f(x,\xi )∥& \le \underset{\tilde{\xi }\in {\mathsf{\Gamma }}_{\mathrm{train}}}{inf}(∥f(x,\xi )-f(x,\tilde{\xi })∥+∥f(x,\tilde{\xi })-{\mathcal{P}}_{\mathcal{V}}f(x,\tilde{\xi })∥\hfill \\ & +∥{\mathcal{P}}_{\mathcal{V}}f(x,\tilde{\xi })-{\mathcal{P}}_{\mathcal{V}}f(x,\xi )∥)\hfill \\ \hfill & \le 2\underset{\tilde{\xi }\in {\mathsf{\Gamma }}_{\mathrm{train}}}{inf}∥f(x,\xi )-f(x,\tilde{\xi })∥+\frac{tol}{3}\hfill \\ \hfill & \le \frac{2tol}{3}+\frac{tol}{3}=tol.\hfill \end{array}$$

(6)

The inequality (6) means that the standard greedy procedure with a fine candidate sample set ${\mathsf{\Gamma }}_{\mathrm{train}}$ can generate a linear approximation space which approximates the target manifold $\mathcal{M}$ with any tolerance $tol$.

2.2. Empirical Interpolation Method (EIM)

Section 2.1 explains that, when the candidate sample set ${\mathsf{\Gamma }}_{\mathrm{train}}$ is fine enough, the greedy procedure can produce a linear approximation space $\mathcal{V}$ which is a $tol$-approximation space for the original target manifold $\mathcal{M}$, i.e.,

$$\underset{f(x,\xi )\in \mathcal{M}}{sup}\parallel f(x,\xi )-{\mathcal{P}}_{\mathcal{V}}f(x,\xi )\parallel \le tol.$$

(7)

For this linear approximation space $\mathcal{V}$, if we denote a set of orthonormal basis functions of $\mathcal{V}$ as $\left\{{\varphi }_1\left(x\right),\cdots ,{\varphi }_n\left(x\right)\right\}$, the orthogonal projection ${\mathcal{P}}_{\mathcal{V}}$ can be expressed as

$${\mathcal{P}}_{\mathcal{V}}f(x,\xi )=\sum _{i=1}^n{\beta }_i\left(\xi \right){\varphi }_i\left(x\right).$$

(8)

Here, ${\beta }_i\left(\xi \right)\in \mathbb{R},i=1,\cdots ,n,$ are the coefficients corresponding to the basis function ${\varphi }_i\left(x\right)$. Since the basis functions ${\varphi }_i\left(x\right),i=1,\cdots ,n$ are orthonormal, the coefficient ${\beta }_i$ can be calculated by

$${\beta }_i\left(\xi \right)=<f(x,\xi ),{\varphi }_i\left(x\right)>={\int }_{\mathsf{\Omega }}f(x,\xi ){\varphi }_i\left(x\right)\mathrm{d}x,\forall i,1\le i\le n.$$

(9)

However, in order to get the coefficient ${\beta }_i\left(\xi \right)$ in (9), we need to evaluate an integration with target function $f(x,\xi )$, which is inconvenient.

The interpolation method [2,18,21] is widely used in function approximation. It approximates the target function $f(x,\xi )$ by restricting the values of this function on n interpolation points $x^{\left(1\right)},\cdots ,x^{\left(n\right)}\in \mathsf{\Omega }$. If we denote the approximation function as ${\mathcal{I}}_{\mathsf{\Omega }}f(x,\xi )\approx f(x,\xi )$, the restrictions are

$${\mathcal{I}}_{\mathsf{\Omega }}f\left(x^{\left(i\right)},\xi \right)=f\left(x^{\left(i\right)},\xi \right),\forall \xi \in \mathsf{\Gamma },\forall i,1\le i\le n.$$

(10)

In addition, if we approximate the target function $f(x,\xi )$ using interpolation methods in a linear approximation space $\mathcal{V}$, and denote it as ${\mathcal{I}}_{\mathcal{V}}f(x,\xi )$, then (8) and (10) can be written as

$${\mathcal{I}}_{\mathcal{V}}f\left(x^{\left(i\right)},\xi \right)=\sum _{i=1}^n{\beta }_i\left(\xi \right){\varphi }_i\left(x^{\left(i\right)}\right)=f\left(x^{\left(i\right)},\xi \right),\forall \xi \in \mathsf{\Gamma },\forall i,1\le i\le n.$$

(11)

Here, ${\beta }_i\left(\xi \right)$, $i=1,\cdots ,n$, are the unknown coefficients. Moreover, we denote the matrix $\mathsf{\Phi }\in {\mathbb{R}}^{n\times n}$ as

$$\mathsf{\Phi }=\left(\begin{array}{ccc}{\varphi }_1\left(x^{\left(1\right)}\right)& \cdots & {\varphi }_n\left(x^{\left(1\right)}\right)\\ ⋮& \ddots & ⋮\\ {\varphi }_1\left(x^{\left(n\right)}\right)& \cdots & {\varphi }_n\left(x^{\left(n\right)}\right)\end{array}\right).$$

Then, the coefficient $\underline{\beta }\left(\xi \right):={\left({\beta }_1\left(\xi \right),\cdots ,{\beta }_n\left(\xi \right)\right)}^T$ satisfying (11) is the solution of the linear system

$$\begin{array}{cc}\hfill \mathsf{\Phi }\underline{\beta }\left(\xi \right)& =\left(\begin{array}{ccc}{\varphi }_1\left(x^{\left(1\right)}\right)& \cdots & {\varphi }_n\left(x^{\left(1\right)}\right)\\ ⋮& \ddots & ⋮\\ {\varphi }_1\left(x^{\left(n\right)}\right)& \cdots & {\varphi }_n\left(x^{\left(n\right)}\right)\end{array}\right)\left(\begin{array}{c}{\beta }_1\left(\xi \right)\\ ⋮\\ {\beta }_n\left(\xi \right)\end{array}\right)\hfill \\ & =\left(\begin{array}{c}\sum _{i=1}^n{\beta }_i\left(\xi \right){\varphi }_i\left(x^{\left(1\right)}\right)\\ ⋮\\ \sum _{i=1}^n{\beta }_i\left(\xi \right){\varphi }_i\left(x^{\left(n\right)}\right)\end{array}\right)=\left(\begin{array}{c}f\left(x^{\left(1\right)},\xi \right)\\ ⋮\\ f\left(x^{\left(n\right)},\xi \right)\end{array}\right).\hfill \end{array}$$

(12)

Compared with (9), $\underline{\beta }$ which is evaluated by the interpolation method (12) avoids the integration and only needs to know the value of target function $f(x,\xi )$ on n interpolation points $x^{\left(1\right)},\cdots ,x^{\left(n\right)}$. The target function $f(x,\xi )$ can then be approximated by

$$f(x,\xi )\approx {\mathcal{I}}_{\mathcal{V}}f(x,\xi )=\sum _{i=1}^n{\beta }_i\left(\xi \right){\varphi }_i\left(x\right),$$

(13)

where the coefficients ${\beta }_i\left(\xi \right),i=1,\cdots ,n$ are the solution of (12). Then, (13) naturally satisfies the restriction (11) by the definition of the coefficients ${\beta }_i\left(\xi \right),i=1,\cdots ,n$.

In EIM [2,3,17,21], systematical approaches to choose suitable interpolation points are given. Here, approximation basis functions and interpolation points are typically obtained alternately by minimizing the error between the target function $f(x,\xi )$ and its approximation function ${\mathcal{I}}_{\mathcal{V}}f(x,\xi )$, of which the produce can be summarized as follows. First, we initialize $V_0=span\left(0\right)$, and select a new parameter value as

$${\xi }^{(k+1)}:={argmax}_{\xi \in {\mathsf{\Gamma }}_{\mathrm{train}}}\parallel f(x,\xi )-{\mathcal{I}}_{{\mathcal{V}}_k}f(x,\xi )\parallel .$$

For the parameter value ${\xi }^{(k+1)}$, if the error

$$\eta \left({\xi }^{(k+1)}\right)=∥f\left(x,{\xi }^{(k+1)}\right)-{\mathcal{I}}_{{\mathcal{V}}_k}f\left(x,{\xi }^{(k+1)}\right)∥$$

(14)

is greater than $tol$, the linear approximation space is updated as ${\mathcal{V}}_{k+1}=$$span\left({\mathcal{V}}_k\cup f\left(x,{\xi }^{(k+1)}\right)\right)$. Then, we orthonormalize the target function $f\left(x,{\xi }^{(k+1)}\right)$ with the basis functions ${\varphi }_1\left(x\right),\cdots ,{\varphi }_k\left(x\right)$ by Schmidt orthogonalization for numerical stability and denote it as ${\varphi }_{k+1}\left(x\right)$, i.e.,

$$\begin{array}{cc}\hfill e\left(x\right)& =f\left(x,{\xi }^{(k+1)}\right)-\sum _{i=1}^k⟨f\left(x,{\xi }^{(k+1)}\right),{\varphi }_i\left(x\right)⟩{\varphi }_i\left(x\right),\hfill \\ \hfill {\varphi }_{k+1}\left(x\right)& =\frac{e\left(x\right)}{\parallel e\left(x\right)\parallel },\hfill \end{array}$$

where $⟨f\left(x,{\xi }^{(k+1)}\right),{\varphi }_i\left(x\right)⟩,i=1,2,\cdots ,k,$ are the coefficients of the projection of $f\left(x,{\xi }^{(k+1)}\right)$ over ${\mathcal{V}}_k=span\left(\left\{{\varphi }_1\left(x\right),{\varphi }_2\left(x\right),\cdots ,{\varphi }_k\left(x\right)\right\}\right)$. The next interpolation point is selected by maximizing the absolute value of the error $r_k\left(x\right)={\varphi }_{k+1}\left(x\right)-{\mathcal{I}}_{{\mathcal{V}}_k}{\varphi }_{k+1}\left(x\right)$, i.e.,

$$x^{(k+1)}={argmax}_{x\in \mathsf{\Omega }}\left|r_k\left(x\right)\right|.$$

The above procedure is repeated until the error $\eta \left(\xi \right)\le tol$ (see (14)) for all $\xi \in {\mathsf{\Gamma }}_{\mathrm{train}}$, and we denote $\mathcal{V}={\mathcal{V}}_n$ for the final step. The relationship between the error $∥f\left(x,\xi \right)-{\mathcal{I}}_{\mathcal{V}}f\left(x,\xi \right)∥$ and $∥f\left(x,\xi \right)-{\mathcal{P}}_{\mathcal{V}}f\left(x,\xi \right)∥$ are discussed in [2] (see [6] for its discrete version).

3. Discrete Empirical Interpolation Method and Its Stochastic Version

We evaluate the values of $f\left(x,{\xi }^{\left(k\right)}\right)$ on a discrete physical domain, that is, computing a vector function

$$\xi \mapsto {\mathbf{f}}_h\left(\xi \right),$$

where ${\mathbf{f}}_h\left(\xi \right)={\left(f\left(x^{\left(1\right)},\xi \right),\cdots ,f\left(x^{\left(N_h\right)},\xi \right)\right)}^T$ with $N_h$ components is the discrete version of $f\left(x,\xi \right)$ on $N_h$ physical points $x^{\left(1\right)},\cdots ,x^{\left(N_h\right)}$. The number of discrete points $N_h$ is usually large to meet certain appropriation accuracy. On the other hand, since the physical domain has been discretized, selecting interpolation points is to find proper indices from $1,2,\cdots ,N_h$. If $i_1,\cdots ,i_n$ denote the indices of the interpolation points and $\mathbf{P}\in {\mathbb{R}}^{N_h\times n}$ denote the matrix of interpolation points defined as

$$\mathbf{P}=[e_{i_1},\cdots ,e_{i_n}]$$

with $e_i\in {\mathbb{R}}^{N_h\times 1}$ the i-th canonical unit vector, the values of target function ${\mathbf{f}}_h\left(\xi \right)$ on the interpolation points can be written as ${\mathbf{P}}^T{\mathbf{f}}_h\left(\xi \right)$. As a result, the corresponding interpolation only needs the values of ${\mathbf{f}}_h\left(\xi \right)$ on $i_1,\cdots ,i_n$ components with $n\ll N_h$.

3.1. Discrete Empirical Interpolation Method (DEIM)

Before presenting our stochastic discrete empirical interpolation method, we here review the original DEIM [6]. In the discrete formulation, a linear approximation space $\mathcal{V}$ can be expressed as

$$\mathcal{V}=span\left(\left\{{\mathbf{f}}_h\left({\xi }^{\left(1\right)}\right),\cdots ,{\mathbf{f}}_h\left({\xi }^{\left(n\right)}\right)\right\}\right),$$

which is an approximation of the discrete target manifold

$${\mathcal{M}}_h=\left\{{\mathbf{f}}_h\left(\xi \right)|\xi \in \mathsf{\Gamma }\right\}.$$

Denoting a set of orthonormal basis functions of this linear approximation space $\mathcal{V}$ by $\left\{{\mathbf{q}}_i\in {\mathbb{R}}^{N_h\times 1},i=1,2,\cdots ,n\right\}$, and denoting the matrix of basis functions by $\mathbf{Q}=[{\mathbf{q}}_1,\cdots ,{\mathbf{q}}_n]\in {\mathbb{R}}^{N_h\times n}$, the approximation of the target function ${\mathbf{f}}_h\left(\xi \right)$ corresponding to (8) can be written as

$${\mathbf{f}}_h\left(\xi \right)\approx {\widehat{\mathbf{f}}}_h:=\mathbf{Q}\underline{\beta }\left(\xi \right)=\sum _{i=1}^n{\beta }_i\left(\xi \right){\mathbf{q}}_i,$$

where $\underline{\beta }\left(\xi \right)={({\beta }_1,\cdots ,{\beta }_n)}^T$ is the coefficient. The restriction of interpolation points corresponding to (13) is

$${\mathbf{P}}^T{\mathbf{f}}_h\left(\xi \right)={\mathbf{P}}^T\mathbf{Q}\underline{\beta }\left(\xi \right),$$

which gives $\underline{\beta }\left(\xi \right)={\left({\mathbf{P}}^T\mathbf{Q}\right)}^{-1}{\mathbf{P}}^T{\mathbf{f}}_h\left(\xi \right)$. Then, it can naturally derive

$${\widehat{\mathbf{f}}}_h\left(\xi \right)=\mathbf{Q}\underline{\beta }\left(\xi \right)=\underset{\tilde{\mathbf{Q}}}{\underbrace{\mathbf{Q}{\left({\mathbf{P}}^T\mathbf{Q}\right)}^{-1}}}\underset{\tilde{\underline{\beta }}\left(\xi \right)}{\underbrace{{\mathbf{P}}^T{\mathbf{f}}_h\left(\xi \right)}}.$$

(15)

By the way $\underline{\beta }$ is obtained, the values of the approximation function ${\widehat{\mathbf{f}}}_h\left(\xi \right)$ are equal to the target function ${\mathbf{f}}_h\left(\xi \right)$ on the components $i_1,\cdots ,i_n$ for any $\xi \in \mathsf{\Gamma }$, i.e., ${\mathbf{P}}^T{\widehat{\mathbf{f}}}_h\left(\xi \right)={\mathbf{P}}^T{\mathbf{f}}_h\left(\xi \right)$. Moreover, the calculation of the new matrix of basis functions $\tilde{\mathbf{Q}}=\mathbf{Q}{\left({\mathbf{P}}^T\mathbf{Q}\right)}^{-1}\in {\mathbb{R}}^{N_h\times n}$ is required once only. For a new realization $\xi$, DEIM only computes the values of the target function ${\mathbf{f}}_h\left(\xi \right)$ on the components $i_1,\cdots ,i_n$ as the new coefficient $\tilde{\underline{\beta }}\left(\xi \right)={\mathbf{P}}^T{\mathbf{f}}_h\left(\xi \right)\in {\mathbb{R}}^{n\times 1}$.

The procedure of DEIM to construct a linear approximation space and select interpolation points can be stated as follows. For a fine candidate sample set ${\mathsf{\Gamma }}_{\mathrm{train}}$, it first chooses

$${\xi }^{\left(1\right)}={argmax}_{\xi \in {\mathsf{\Gamma }}_{\mathrm{train}}}{∥{\mathbf{f}}_h\left(\xi \right)∥}_2,$$

where ${\parallel ·\parallel }_2$ is the vector $L_2$ norm for a discrete function. Then, we orthonormalize ${\mathbf{f}}_h\left({\xi }^{\left(1\right)}\right)$ as ${\mathbf{q}}_1$ and denote ${\mathbf{Q}}_1=\left[{\mathbf{q}}_1\right]$ with

$${\mathbf{q}}_1:=\frac{{\mathbf{f}}_h\left({\xi }^{\left(1\right)}\right)}{{∥{\mathbf{f}}_h\left({\xi }^{\left(1\right)}\right)∥}_2}.$$

The index of the first interpolation point $i_1$ is initialized as

$$i_1={argmax}_{i=1,\cdots ,N_h}\left|e_i^T{\mathbf{q}}_1\right|,$$

(16)

and we let ${\mathbf{P}}_1=\left[e_{i_1}\right]$. For the $(k+1)$-th basis function and interpolation point, a parameter value ${\xi }^{(k+1)}$ is chosen from the candidate sample set ${\mathsf{\Gamma }}_{\mathrm{train}}$ by finding the maximum error between the target function ${\mathbf{f}}_h\left(\xi \right)$ and its approximation ${\widehat{\mathbf{f}}}_h\left(\xi \right)$ in the linear approximation space ${\mathcal{V}}_k=span\left(\{{\mathbf{q}}_1,\cdots ,{\mathbf{q}}_k\}\right)$, i.e.,

$${\xi }^{(k+1)}={argmax}_{\xi \in {\mathsf{\Gamma }}_{\mathrm{train}}}{∥{\mathbf{f}}_h\left(\xi \right)-{\widehat{\mathbf{f}}}_h\left(\xi \right)∥}_2.$$

(17)

If the error $\eta \left({\xi }^{(k+1)}\right)={∥{\mathbf{f}}_h\left({\xi }^{(k+1)}\right)-{\widehat{\mathbf{f}}}_h\left({\xi }^{(k+1)}\right)∥}_2$ is greater than a given tolerance $tol$, we orthonormalize ${\mathbf{f}}_h\left({\xi }^{(k+1)}\right)$ with ${\mathbf{q}}_1,\cdots ,{\mathbf{q}}_k$ by Schmidt orthogonalization and denote it as ${\mathbf{q}}_{k+1}$, that is,

$$\begin{array}{cc}\hfill {\mathbf{e}}^{(k+1)}& :={\mathbf{f}}_h\left({\xi }^{(k+1)}\right)-{\mathbf{Q}}_k{\mathbf{Q}}_k^T{\mathbf{f}}_h\left({\xi }^{(k+1)}\right),\hfill \\ \hfill {\mathbf{q}}^{(k+1)}& :=\frac{{\mathbf{e}}^{(k+1)}}{\parallel {\mathbf{e}}^{(k+1)}{\parallel }_2}.\hfill \end{array}$$

(18)

Then, the matrix of basis functions is updated as ${\mathbf{Q}}_{k+1}=[{\mathbf{Q}}_k,{\mathbf{q}}_{k+1}]$. The residual of interpolation projecting ${\mathbf{q}}_{k+1}$ to the linear approximation space ${\mathcal{V}}_k$ can be written as

$${\mathbf{r}}_k:={\mathbf{q}}_{k+1}-{\widehat{\mathbf{q}}}_{k+1}={\mathbf{q}}_{k+1}-{\mathbf{Q}}_k{\left({\mathbf{P}}_k^T{\mathbf{Q}}_k\right)}^{-1}{\mathbf{P}}_k^T{\mathbf{q}}_{k+1}.$$

(19)

An index of $(k+1)$-th interpolation point $i_{k+1}$ is selected by finding the maximum component of residual ${\mathbf{r}}_k$, i.e.,

$$i_{k+1}={argmax}_{i=1,\cdots ,N_h}\left|e_i^T{\mathbf{r}}_k\right|,$$

(20)

and we update ${\mathbf{P}}_{k+1}=[{\mathbf{P}}_k,e_{i_{k+1}}]$. Finally, we denote $\mathcal{V}={\mathcal{V}}_n$, $\mathbf{Q}={\mathbf{Q}}_n$ and $\mathbf{P}={\mathbf{P}}_n$. This linear approximation space $\mathcal{V}$ then satisfies

$$\underset{\xi \in {\mathsf{\Gamma }}_{\mathrm{train}}}{sup}{∥{\mathbf{f}}_h\left(\xi \right)-{\widehat{\mathbf{f}}}_h\left(\xi \right)∥}_2\le tol.$$

The whole procedure is stated in Algorithm 1, where $\eta$ means the error in (14) or its discrete form (17).

Algorithm 1 Discrete empirical interpolation method (DEIM) [6]

Input: A candidate sample set ${\mathsf{\Gamma }}_{\mathrm{train}}$ and a target function ${\mathbf{f}}_h\left(\xi \right)$.

1: Initialize ${\xi }^{\left(1\right)}={argmax}_{\xi \in {\mathsf{\Gamma }}_{\mathrm{train}}}{∥{\mathbf{f}}_h\left(\xi \right)∥}_2$ and orthonormalize ${\mathbf{f}}_h\left({\xi }^{\left(1\right)}\right)$ as ${\mathbf{q}}_1={\mathbf{f}}_h\left({\xi }^{\left(1\right)}\right)/{∥{\mathbf{f}}_h\left({\xi }^{\left(1\right)}\right)∥}_2$.

2: Initialize $i_1={argmax}_{i=1,\cdots ,N_h}\left|e_i^T{\mathbf{q}}_1\right|$.

3: Initialize $\mathbf{Q}=\left[{\mathbf{q}}_1\right]$ and $\mathbf{P}=\left[e_{i_1}\right]$.

4: while ${sup}_{\xi \in {\mathsf{\Gamma }}_{\mathrm{train}}}\eta \left(\xi \right)={∥{\mathbf{f}}_h\left(\xi \right)-{\widehat{\mathbf{f}}}_h\left(\xi \right)∥}_2>tol$ do

5:   Compute the error $\eta \left({\xi }^{\left(i\right)}\right)$ for ${\xi }^{\left(i\right)}\in {\mathsf{\Gamma }}_{\mathrm{train}}$, $i=1,2,\cdots ,|{\mathsf{\Gamma }}_{\mathrm{train}}|$.

6:   Let ${\xi }^{(k+1)}={argmax}_{\xi \in {\mathsf{\Gamma }}_{\mathrm{train}}}\eta \left(\xi \right)$.

7:   Compute ${\mathbf{q}}_{k+1}$ through orthonormalizing ${\mathbf{f}}_h\left({\xi }^{(k+1)}\right)$ with ${\mathbf{q}}_1,\cdots ,{\mathbf{q}}_k$ by (18).

8:   Solve the equation ${\mathbf{P}}^T{\mathbf{q}}_{k+1}={\mathbf{P}}^T\mathbf{Q}\underline{\beta }$ for $\underline{\beta }$.

9:   Compute the residual ${\mathbf{r}}_k={\mathbf{q}}_{k+1}-{\widehat{\mathbf{q}}}_{k+1}={\mathbf{q}}_{k+1}-\mathbf{Q}\underline{\beta }$.

10:    Select the interpolation index $i_{k+1}$ as $i_{k+1}={argmax}_{i=1,\cdots ,N_h}\left|e_i^T{\mathbf{r}}_k\right|$.

11:    Update $\mathbf{Q}=[\mathbf{Q},{\mathbf{q}}_{k+1}]$ and $\mathbf{P}=[\mathbf{P},{\mathbf{e}}_{i_{k+1}}]$.

12: end while

Output: The matrix of basis functions $\mathbf{Q}$ and the matrix of interpolation points $\mathbf{P}$.

3.2. Stochastic Discrete Empirical Interpolation Method (SDEIM)

For standard EIM (or DEIM), to satisfy the $tol$-approximation (see (7)), the candidate sample set ${\mathsf{\Gamma }}_{\mathrm{train}}$ typically needs to be fine enough (i.e., the condition (5) holds). However, if the size of ${\mathsf{\Gamma }}_{\mathrm{train}}$ is large, it can be expensive to construct the DEIM approximation (15). In this section, we propose a stochastic discrete empirical interpolation method (SDEIM). In SDEIM, we accept the approximation with probability instead of giving a threshold with certainty, and we then avoid fine candidate sample sets. We note that a weight EIM strategy is developed in [4], while the purpose of our work is to give a stochastic criterion for updating training sets.

Our problem formulation is still to approximate the target manifold ${\mathcal{M}}_h$ with a linear approximation space $\mathcal{V}$, such that it satisfies

$$\eta \left(\xi \right)=\parallel {\mathbf{f}}_h\left(\xi \right)-{\mathcal{I}}_{\mathcal{V}}{\mathbf{f}}_h{\left(\xi \right)\parallel }_2\le tol.$$

However, assuming that a probability measure $\mathbb{P}$ exists in the parameter space $\mathsf{\Gamma }$, we herein do not ensure $\eta \left(\xi \right)\le tol$ for all $\xi \in \mathsf{\Gamma }$, but concern about the failure probability

$$p:=\mathbb{P}\left(\xi \in \mathsf{\Gamma }\left|\eta \right(\xi )>tol\right),$$

(21)

which measures the size of the parameter set where the approximation is not accurate enough. While the probability p can hardly be exactly evaluated, we evaluate its empirical probability among N samples

$$\overline{p}:=\frac{1}{N}\sum _{i=1}^N\mathbb{I}\left(\eta \left({\xi }^{\left(i\right)}\right)-tol\right),$$

(22)

where $\mathbb{I}\left(x\right)$ is the indicator function defined as

$$\mathbb{I}\left(x\right)=\left\{\begin{array}{cc}1,\hfill & \mathrm{if}x>0,\hfill \\ 0,\hfill & \mathrm{if}x\le 0.\hfill \end{array}\right.$$

The probability (22) is to calculate the average number of occurrences of $\eta \left(\xi \right)>tol$ on N samples. By the law of large numbers [22], the empirical probability $\overline{p}$ converges to the probability p with probability one as N goes to infinite, i.e.,

$$\mathbb{P}\left(\underset{N\to \infty }{lim}\frac{1}{N}\sum _{i=1}^N\mathbb{I}\left(\eta \left({\xi }^{\left(i\right)}\right)-tol\right)=p\right)=1.$$

Since the implicit constant p reflects the probability that $\xi$ does not satisfy the tolerance $tol$, we hope that p is small enough. On the other hand, p can not be evaluated explicitly, and the empirical probability $\overline{p}$ is an approximation of p. In SDEIM, therefore, $\overline{p}$ is set to be small enough. For convenience, we take $\overline{p}=0$ in the verifying stage and use the sample size N to control the accuracy of $\overline{p}$ to approximate p.

The procedure of SDEIM has two steps: constructing a linear approximation space $\mathcal{V}$, and verifying whether the empirical probability $\overline{p}=0$ for this linear approximation space $\mathcal{V}$ in N consecutive samples. Our SDEIM algorithm can be described as follows. First, a sample ${\xi }^{\left(1\right)}\in \mathsf{\Gamma }$ is randomly selected, and the target function ${\mathbf{f}}_h\left({\xi }^{\left(1\right)}\right)$ is normalized as ${\mathbf{q}}_1$. The linear approximation space is initialized as ${\mathcal{V}}_1=span\left(\left\{{\mathbf{q}}_1\right\}\right)$ and the matrix of basis function is denoted as ${\mathbf{Q}}_1=\left[{\mathbf{q}}_1\right]$. The index of the first interpolation point $i_1$ is initialized in the same way as (16) and the matrix of interpolation point is denoted as ${\mathbf{P}}_1=\left[e_{i_1}\right]$. Then, the empirical probability $\overline{p}$ is verified for this linear approximation space ${\mathcal{V}}_1$. We sample N consecutive samples. If one of these N samples makes $\eta \left(\xi \right)>tol$, we orthonormalize ${\mathbf{q}}_{k+1}$ for this sample $\xi$, find the index of the $(k+1)$-th interpolation point $i_{k+1}$ as DEIM in (19) and (20) and update ${\mathbf{P}}_{k+1}=[{\mathbf{P}}_k,e_{i_{k+1}}]$. The linear approximation space is updated as ${\mathcal{V}}_{k+1}=span\left({\mathcal{V}}_k\cup {\mathbf{q}}_{k+1}\right)$ and the matrix of the basis function is updated as ${\mathbf{Q}}_{k+1}=[{\mathbf{Q}}_k,{\mathbf{q}}_{k+1}]$. Then, the empirical probability $\overline{p}$ is verified again for this linear approximation space ${\mathcal{V}}_{k+1}$. The above steps are repeated alternately until there is a linear approximation space ${\mathcal{V}}_n$ such that $\eta \left({\xi }^{\left(i\right)}\right)\le tol$ for N consecutive samples. Finally, a linear approximation space $\mathcal{V}={\mathcal{V}}_n$ is obtained, such that the empirical probability $\overline{p}=0$ for N consecutive samples. Details of SDEIM are stated as Algorithm 2.

Algorithm 2 Stochastic discrete empirical interpolation method (SDEIM)

Input: A constant number N and a target function ${\mathbf{f}}_h\left(\xi \right)$.

1: Sample $\xi$ randomly.

2: Evaluate the target function ${\mathbf{f}}_h\left(\xi \right)$ and initialize ${\mathbf{q}}_1={\mathbf{f}}_h\left(\xi \right)/{∥{\mathbf{f}}_h\left(\xi \right)∥}_2$.

3: Initialize $i_1={argmax}_{i=1,\cdots ,N_h}\left|e_i^T{\mathbf{q}}_1\right|$.

4: Initialize $\mathbf{Q}=\left[{\mathbf{q}}_1\right]$ and $\mathbf{P}=\left[{\mathbf{e}}_{i_1}\right]$.

5: Initialize the counting index $j=0$.

6: while $j\le N$ (if $j>N$, it means that $\overline{p}=0$ in verifying stage) do

7:  Sample $\xi \in \mathsf{\Gamma }$ randomly.

8:  if error $\eta \left(\xi \right)={∥{\mathbf{f}}_h\left(\xi \right)-{\widehat{\mathbf{f}}}_h\left(\xi \right)∥}_2>tol$ then

9:      Set counting index $j=0$.

10:    Compute ${\mathbf{q}}_{k+1}$ through orthonormalizing ${\mathbf{f}}_h\left(\xi \right)$ with the ${\mathbf{q}}_1,\cdots ,{\mathbf{q}}_k$ by (18).

11:    Solve ${\mathbf{P}}^T{\mathbf{q}}_{k+1}={\mathbf{P}}^T\mathbf{Q}\underline{\beta }$ for $\underline{\beta }$.

12:    Compute the residual ${\mathbf{r}}_k={\mathbf{q}}_{k+1}-{\widehat{\mathbf{q}}}_{k+1}={\mathbf{q}}_{k+1}-\mathbf{Q}\underline{\beta }$.

13:    Select the interpolation index $i_{k+1}$ as $i_{k+1}={argmax}_{i=1,\cdots ,N_h}\left|e_i^T{\mathbf{r}}_k\right|$.

14:    Update $\mathbf{Q}=[\mathbf{Q},{\mathbf{q}}_{k+1}]$ and $\mathbf{P}=[\mathbf{P},{\mathbf{e}}_{i_{k+1}}]$.

15:  else

16:    Update $j=j+1$.

17:  end if

18: end while

Output: The matrix of basis functions $\mathbf{Q}$ and the matrix of interpolation points $\mathbf{P}$.

3.3. Performance and Complexity of SDEIM

In SDEIM, we ensure that the empirical probability $\overline{p}=0$ for N consecutive samples in the verifying stage, i.e., the number of $\xi$ such that $\eta \left(\xi \right)>tol$ is zero in these N consecutive samples. The probability p is adjusted through the samples size N, which affects the accuracy of $\overline{p}$ for approximating p. For any tolerance $tol$ and sample size N, there is always a linear approximation space $\mathcal{V}$ such that $\eta \left(\xi \right)=\parallel {\mathbf{f}}_h\left(\xi \right)-{\mathcal{I}}_{\mathcal{V}}{\mathbf{f}}_h{\left(\xi \right)\parallel }_2\le tol$ for N consecutive samples—in the worst case, when the dimension of the linear approximation space $dim\left(\mathcal{V}\right)=N_h$, this linear approximation space $\mathcal{V}$ obviously satisfies the condition.

Although the probability value p cannot be evaluated explicitly, the empirical probability $\overline{p}$ approximates the probability p with probability one as N goes to infinite. Hence, for a given threshold $\epsilon$ and a confidence $(1-\delta )$, we can consider this question, whether there is

$$\mathbb{P}\left(|p-\overline{p}|<\epsilon \right)\ge 1-\delta .$$

(23)

By the law of large numbers, the answer is always correct for a suitable N. Before describing the relationship between N and the probability p, the threshold $\epsilon$ and the confidence level $(1-\delta )$, we first introduce the Hoeffding’s inequality as Lemma 1.

Lemma 1

(Hoeffding’s inequality [23]). Let random variables $X_1,X_2,\cdots ,X_N$ be independent and identically distributed with values in the interval $[0,1]$ and expectation $EX$, then for any $\epsilon >0$

$$\begin{array}{cc}\hfill \mathbb{P}\left(\left|\mathbb{E}X-\frac{1}{N}\sum _{i=1}^N{X}_i\right|\ge \epsilon \right)& \le 2exp\left(-2N{\epsilon }^2\right),\hfill \\ \hfill \mathbb{P}\left(\mathbb{E}X-\frac{1}{N}\sum _{i=1}^N{X}_i\ge \epsilon \right)& \le exp\left(-2N{\epsilon }^2\right).\hfill \end{array}$$

The Hoeffding’s inequality characters the relationship between the arithmetic mean $\frac{1}{N}{\sum }_{i=1}^N{X}_i$ and its expectation $\mathbb{E}X$ for bounded random variables. In SDEIM, when a linear approximation space $\mathcal{V}$ is given, the error $\eta \left(\xi \right)$ can only be greater than tolerance $tol$ or not greater than $tol$ for any realization $\xi$. Hence, the indicator $\mathbb{I}\left(\eta \left(\xi \right)-tol\right)$ is a random variable taking value of zero or one with expectation $\mathbb{E}\mathbb{I}\left(\eta \left(\xi \right)-tol\right)=p$. Note that we set the empirical probability $\overline{p}=0$ in the verifying stage. Then, the problem (23) can be answered with Theorem 1.

Theorem 1.

For any significant level δ ($0<\delta <1$) and threshold ε, the linear approximation space $\mathcal{V}$ is produced by SDEIM with sample size N. Then, if $N\ge \frac{1}{2{\epsilon }^2}ln\frac{1}{\delta }$, we have

$$\mathbb{P}(p<\epsilon )\ge 1-\delta .$$

Proof. 

By Lemma 1 and $\overline{p}=0$ in SDEIM, we have

$$\begin{array}{cc}\hfill \mathbb{P}(p<\epsilon )& =1-\mathbb{P}(p\ge \epsilon )\hfill \\ \hfill & =1-\mathbb{P}(p-\overline{p}\ge \epsilon )\hfill \\ \hfill & \ge 1-exp(-2N{\epsilon }^2)\hfill \\ \hfill & \ge 1-exp(-2{\epsilon }^2\frac{1}{2{\epsilon }^2}ln\frac{1}{\delta })\hfill \\ \hfill & =1-\delta .\hfill \end{array}$$

□

Moreover, the relationship between probability p and the sample size N, the threshold $\epsilon$, the confidence $(1-\delta )$, can be explicitly described as Theorem 2.

Theorem 2.

For any significance level δ ($0<\delta <1$), the linear approximation space $\mathcal{V}$ is produced by SDEIM with sample size N. There is at least confidence $(1-\delta )$ satisfying

$$0<p<\sqrt{\frac{1}{2N}ln\frac{1}{\delta }}.$$

It means that, if the error estimator in SDEIM is set to $\eta \left(\xi \right)=\parallel {\mathbf{f}}_h\left(\xi \right)-{\mathcal{I}}_{\mathcal{V}}{\mathbf{f}}_h{\left(\xi \right)\parallel }_2$, with confidence $(1-\delta )$, we have

$$\mathbb{P}\left(\parallel {\mathbf{f}}_h\left(\xi \right)-{\mathcal{I}}_{\mathcal{V}}{\mathbf{f}}_h\left(\xi \right){\parallel }_2>tol\right)<\sqrt{\frac{1}{2N}ln\frac{1}{\delta }}$$

(24)

and

$$\mathbb{P}\left(\parallel {\mathbf{f}}_h\left(\xi \right)-{\mathcal{I}}_{\mathcal{V}}{\mathbf{f}}_h\left(\xi \right){\parallel }_2\le tol\right)\ge 1-\sqrt{\frac{1}{2N}ln\frac{1}{\delta }}.$$

(25)

Proof. 

Without loss of generality, we set

$$\delta =exp(-2N{\epsilon }^2)$$

for suitable ε, that is

$$\epsilon =\sqrt{\frac{1}{2N}ln\frac{1}{\delta }}.$$

By Lemma 1 and $\overline{p}=0$ in SDEIM, we have

$$\mathbb{P}\left(p\ge \sqrt{\frac{1}{2N}ln\frac{1}{\delta }}\right)\le exp(-2N{\epsilon }^2)=\delta .$$

That is

$$p<\sqrt{\frac{1}{2N}ln\frac{1}{\delta }}$$

with confidence $(1-\delta )$. In addition, when the error estimator is set to $\eta \left(\xi \right)=\parallel {\mathbf{f}}_h\left(\xi \right)-{\mathcal{I}}_{\mathcal{V}}{\mathbf{f}}_h{\left(\xi \right)\parallel }_2$, since the definition of p in (21) is

$$p=\mathbb{P}\left(\xi \in \mathsf{\Gamma }\left|\eta \right(\xi )>tol\right),$$

(24) and (25) can be derived directly. □

Theorem 2 ensures that the linear approximation space $\mathcal{V}$ given by the SDEIM algorithm (Agorithm 2), can approximate the target function ${\mathbf{f}}_h\left(\xi \right)$ with the tolerance $tol$ and the probability threshold $\epsilon$. Actually, in Algorithm 2, we reset the counting index each time $\eta \left(\xi \right)>tol$ occurs, which means that there are more than N samples such that $\eta \left(\xi \right)<tol$. Hence, $\sqrt{\frac{1}{2N}ln\frac{1}{\delta }}$ is a more strict upper bound.

In the procedure of Algorithm 2, the best situation is that n consecutive samples are compared to generate the linear approximation space $\mathcal{V}$ with dimension $dim\left(\mathcal{V}\right)=n$ and then N samples are compared to verify the error $\eta \left(\xi \right)\le tol$. In this case, the number of comparisons is $\mathcal{O}(n+N)$. The worst situation is that, each sample $\xi$ is found with $\eta \left(\xi \right)>tol$ at the end of the N comparisons for verification and this procedure is repeated n times. It is clear that, the number of comparisons in SDEIM is much smaller than that in standard DEIM, where large training sets typically need to be looped over.

4. Numerical Experiments

In this section, four test problems are considered to show the efficiency of SDEIM. The first one is a nonlinear parameterized function with spatial points in one dimension. The second one is to extend the first experiment to two dimensions. The third one focuses on the property of SDEIM for random fields. The last experiment is a nonlinear ordinary differential equation arising in neuron modeling.

4.1. A Nonlinear Parameterized Function with Spatial Points in One Dimension

Consider a nonlinear parameterized function $f:\mathsf{\Omega }\times \mathsf{\Gamma }\to \mathbb{R}$ defined by

$$f(x,\xi )=10xsin\left(2\pi \xi x\right),$$

where $x\in \mathsf{\Omega }=[0,1]\subset \mathbb{R}$ and $\xi \in \mathsf{\Gamma }=[1,T]\subset \mathbb{R}$ for a constant T. For a given parameter $\xi$, the function $f(x,\xi )$ has the period $1/\xi$. Figure 1 plots the function $f(x,\xi )$ for different $\xi =1,2$ and 8. It shows that the target function can have different complexities for different parameter ranges. Let $\left\{x^{\left(i\right)},i=1,2,\cdots ,N_h\right\}$ be a uniform grid points in $\mathsf{\Omega }$ for $N_h=400$, and define ${\mathbf{f}}_h\left(\xi \right):\mathsf{\Gamma }\to {\mathbb{R}}^{N_h}$ as

$${\mathbf{f}}_h\left(\xi \right)={\left[f\left(x^{\left(1\right)},\xi \right),\cdots ,f\left(x^{\left(N_h\right)},\xi \right)\right]}^T\in {\mathbb{R}}^{N_h}$$

for $\xi \in \mathsf{\Gamma }$. Let the range of the parameter be $\mathsf{\Gamma }=[1,T]$ for $T=2$ and let ${\mathsf{\Gamma }}_{\mathrm{train}}$ be selected uniformly over $\mathsf{\Gamma }$ with $|{\mathsf{\Gamma }}_{\mathrm{train}}|=50$ for DEIM. The tolerance $tol$ is set to $tol={10}^{-4}$ and the confidence is set to $0.99$ ($\delta ={10}^{-2}$) for Algorithms 1 and 2.

Figure 1. The parameterized function $f(x,\xi )$ for $\xi =1,2$ and 8.

Figure 2 shows the training procedures for different methods. Figure 2a is the training procedure of SDEIM with threshold $\epsilon =0.70$. An epoch in Figure 2a is a training procedure for a new realization $\xi$. The black curve indicates that the error changes in the training procedure. The green point is the first successful sample with $\eta \left(\xi \right)\le tol$. In DEIM, after traversing the candidate sample set, if the sample with the largest error is less than tolerance $tol$, the algorithm stops and the linear approximation space satisfies $\eta \left(\xi \right)\le tol$ for all $\xi \in {\mathsf{\Gamma }}_{\mathrm{train}}$. In SDEIM, we continue to find whether there are failed samples (the red crosses) in $\mathsf{\Gamma }$ with $\eta \left(\xi \right)>tol$. The SDEIM algorithm (Algorithm 2) stops when the samples satisfy $\eta \left(\xi \right)\le tol$ for N consecutive samples. In this case, the probability for the appearance of a failed sample is considered to be very low in SDEIM. Figure 2b shows the relationship between the number of comparisons and the average error in the training procedure for SDEIM (with $p<0.70$ and $p<0.10$) and DEIM, where the average error is computed on $\tilde{N}={10}^4$ samples defined as

$$\overline{\eta }:=\frac{1}{\tilde{N}}\sum _{i=1}^{\tilde{N}}\eta \left({\xi }^{\left(i\right)}\right).$$

(26)

Figure 2. (a) The training procedure of SDEIM with probability threshold $\epsilon =0.70$, i.e., $p<0.70$, where each epoch is to test a new sample $\xi$ for updating the basis. (b) The relationship between the number of comparisons and the average error in the training procedure for SDEIM (with $p<0.70$ and $p<0.10$) and DEIM.

Since a basis function is found after searching the candidate sample set ${\mathsf{\Gamma }}_{\mathrm{train}}$ in DEIM, its average error decreases in a ladder shape. In SDEIM, errors for an extra N consecutive samples need to be computed, and its numbers of comparisons have a long flat tail.

Table 1 shows more details for the linear approximation space $\mathcal{V}$ produced by the two methods. The average errors $\overline{\eta }$ for different methods are computed using (26) with $\tilde{N}={10}^4$ samples and the empirical probabilities $\overline{p}$ are calculated using (22) with the same $\tilde{N}={10}^4$ samples which are the approximation of p. The number of comparisons is the times in different methods to compare the error $\eta \left(\xi \right)$. From Table 1, it can be seen that since SDEIM does not need to search the sample $\xi$ in a candidate sample set ${\mathsf{\Gamma }}_{\mathrm{train}}$, it has fewer comparisons.

Table 1. The average error $\overline{\eta }$, the number of basis functions n for the approximation space $\mathcal{V}$, the empirical probability $\overline{p}$ and the number of comparisons for SDEIM (with $p<0.70$ and $p<0.10$) and DEIM (with $|{\mathsf{\Gamma }}_{\mathrm{train}}|=50$).

		$\overline{\mathit{\eta }}$	n	$\overline{\mathit{p}}$	Number of Comparisons
SDEIM	$p<0.70$	$2.8815\times {10}^{-5}$	9	$7.36\times {10}^{-2}$	$14+N(N=5)$
	$p<0.10$	$1.0145\times {10}^{-7}$	10	0	$22+N(N=231)$
DEIM	$|{\mathsf{\Gamma }}_{\mathrm{train}}|=50$	$1.9514\times {10}^{-8}$	10	0	550

Figure 3a,b show the first six basis functions (i.e., the first six columns of matrix $\tilde{\mathbf{Q}}$) for DEIM and SDEIM. Figure 3c shows the samples $\xi$ which generate the basis functions for DEIM and SDEIM. The black dots in Figure 3c are the samples in the candidate sample set ${\mathsf{\Gamma }}_{\mathrm{train}}$, and the black dots circled by the blue circles are the samples selected by DEIM. The numbers above them are the order in which they are selected. The red stars are the samples selected by SDEIM, which are generated in random order. Note that the consecutive samples in DEIM are separated further because they are selected after searching the fine candidate sample set ${\mathsf{\Gamma }}_{\mathrm{train}}$.

Figure 3. (a) The first six basis functions for DEIM. (b) The first six basis functions for SDEIM. (c) The samples for generating the basis functions.

4.2. A Nonlinear Parameterized Function with Spatial Points in Two Dimensions

Consider a nonlinear parameterized function $f:\mathsf{\Omega }\times \mathsf{\Gamma }\to \mathbb{R}$ defined by

$$f(x,\xi )=10x_1{x}_{2}sin\left(2\pi {\xi }_1{x}_1\right)cos\left(2\pi {\xi }_2{x}_2\right),$$

where $x=(x_1,x_2)\in \mathsf{\Omega }={[0,1]}^2\subset {\mathbb{R}}^2$ and $\xi =({\xi }_1,{\xi }_2)\in \mathsf{\Gamma }={[1,T]}^2\subset {\mathbb{R}}^2$ for a constant T. Similarly to Section 4.1, for a large ${\xi }_1$ (or ${\xi }_2$), $f(x,\xi )$ changes more quickly in the $x_1$ (or $x_2$) direction. Let $\left\{x^{\left(i\right)},i=1,2,\cdots ,N_h\right\}$ be a uniform grid in $\mathsf{\Omega }$ for $N_h=50\times 50=2500$, and define ${\mathbf{f}}_h\left(\xi \right):\mathsf{\Gamma }\to {\mathbb{R}}^{N_h}$ as

$${\mathbf{f}}_h\left(\xi \right)={\left[f\left(x^{\left(1\right)},\xi \right),\cdots ,f\left(x^{\left(N_h\right)},\xi \right)\right]}^T\in {\mathbb{R}}^{N_h}$$

for $\xi \in \mathsf{\Gamma }$. Let the range of the parameter be $\mathsf{\Gamma }={[1,T]}^2$ for $T=4$. Figure 4 shows the image of $f(x,\xi )$ for $\xi =[4,4]$. The tolerance $tol$ is set to $tol={10}^{-4}$ and the confidence is set to $0.99$ for $\delta ={10}^{-2}$ for Algorithms 1 and 2.

Figure 4. The parameterized function $f(x,\xi )$ for $\xi =[4,4]$.

Figure 5 shows the training procedures for the two methods. Where Figure 5a is the training procedure of SDEIM with threshold $\epsilon =0.10$. An epoch in Figure 5a is a comparison of the error for a new realization of $\xi$. The trend of black dots is the trend of errors, which decrease rapidly at early epochs. For these early epochs, the linear approximation space is updated for each realization $\xi$ in this stage. After the first successful sample (the green point with $\eta \left(\xi \right)\le tol$), the error becomes relatively stable near the $tol$. Then, for SDEIM, the linear approximation space is updated only for the failed samples (the red crosses with $\eta \left(\xi \right)>tol$), and Algorithm 2 stops when the samples satisfy $\eta \left(\xi \right)\le tol$ for N consecutive samples. Figure 5b shows the relationship between the number of comparisons and the average error among $\tilde{N}={10}^4$ samples in the training procedure for SDEIM and DEIM. Compared with DEIM, SDEIM has fewer comparisons.

Figure 5. (a) The training procedure of SDEIM with probability threshold $\epsilon =0.10$, i.e., $p<0.10$, where each epoch is to test a new sample $\xi$ for updating the basis. (b) The relationship between the number of comparisons and the average error in the training procedure for DEIM and SDEIM with different parameters.

Table 2 shows more details for DEIM and SDEIM with different parameters, where the average errors $\overline{\eta }$ and the empirical probability $\overline{p}$ for different parameters are computed using (22) and (26) on the same $\tilde{N}={10}^4$ samples. It can be seen that the empirical probability $\overline{p}$ for DEIM is similar to that of SDEIM. Moreover, by comparing DEIM with the size of candidate sample set $|{\mathsf{\Gamma }}_{\mathrm{train}}|=400$ and SDEIM with probability threshold $\epsilon =0.10$, the number of comparisons for SDEIM is far less than DEIM.

Table 2. The average error $\overline{\eta }$, the number of basis functions n for the approximation space $\mathcal{V}$, the empirical probability $\overline{p}$ and the number of comparisons for SDEIM with ($p<0.70$ and $p<0.10$) and DEIM with ($|{\mathsf{\Gamma }}_{\mathrm{train}}|=225$ and $|{\mathsf{\Gamma }}_{\mathrm{train}}|=400$).

		$\overline{\mathit{\eta }}$	n	$\overline{\mathit{p}}$	Number of Comparisons
SDEIM	$p<0.70$	$3.7000\times {10}^{-3}$	173	$2.631\times {10}^{-1}$	$186+N(N=5)$
	$p<0.10$	$4.0844\times {10}^{-6}$	204	$2.500\times {10}^{-3}$	$869+N(N=231)$
DEIM	$|{\mathsf{\Gamma }}_{\mathrm{train}}|=225$	$1.1881\times {10}^{-4}$	177	$1.876\times {10}^{-1}$	40,050
	$|{\mathsf{\Gamma }}_{\mathrm{train}}|=400$	$2.1180\times {10}^{-5}$	184	$2.520\times {10}^{-2}$	74,000

4.3. Random Fields

Consider a nonlinear parameterized function $f:\mathsf{\Omega }\times \mathsf{\Gamma }\to \mathbb{R}$ defined by

$$f(x,\xi )=exp\left(z(x,\xi )\right),$$

where $x=(x_1,x_2)\in \mathsf{\Omega }={[0,1]}^2\subset {\mathbb{R}}^2$ and $z(x,\xi )$ is assumed to be a stochastic process with mean function $z_0\left(x\right)\equiv 1$ and covariance function $Cov(x,y)$ defined as

$$Cov(x,y)={\sigma }^{2}exp\left(-\frac{|x_1-y_1|}{L}-\frac{|x_2-y_2|}{L}\right).$$

(27)

Here $x=(x_1,x_2)\in \mathsf{\Omega },y=(y_1,y_2)\in \mathsf{\Omega }$ and L is the correlation length. The Karhu nen–Loève (KL) expansion (see [24,25] for details) gives a representation of $z(x,\xi )$ as

$$z(x,\xi )=z_0\left(x\right)+\sum _{i=1}^{\infty }\sqrt{{\lambda }_i}{\phi }_i\left(x\right){\xi }_i,$$

where $\{{\phi }_i\left(x\right),i=1,2,\cdots \}$ are the orthonormal eigenfunctions and $\{{\lambda }_i,i=1,2,\cdots \}$ are the corresponding eigenvalues of the covariance function $Cov(x,y)$. Moreover, $\{{\xi }_i,i=1,2,\cdots \}$ are mutually uncorrelated random variables. In this example, we truncate the expansion to M terms as our surrogate model according the retaining of the energy for $z(x,\xi )$:

$$z_M(x,\xi ):=z_0\left(x\right)+\sum _{i=1}^M\sqrt{{\lambda }_i}{\phi }_i\left(x\right){\xi }_i,$$

where $\xi =({\xi }_1,\cdots ,{\xi }_M)$ is the parameter in the surrogate model whose distribution is assumed to be a uniform distribution in $\mathsf{\Gamma }={[-1,1]}^M\subset {\mathbb{R}}^M$ and M satisfies

$${\delta }_{KL}:=\frac{{\sum }_{i=1}^M{\lambda }_i}{\left|\mathsf{\Omega }\right|{\sigma }^2}>0.95.$$

Here, $\left|\mathsf{\Omega }\right|$ is the area of the physical domain $\mathsf{\Omega }$ and the standard deviation $\sigma$ is set to $\sigma =0.5$ for (27). We focus on the cases $L=1$ and $L=0.5$ in this experiment, and the corresponding dimensions of the parameter spaces are $M=33$ and $M=109$ respectively. The physical grid is set to a uniform grid with $N_h=50\times 50=2500$. We consider the nonlinear parameterized function ${\mathbf{f}}_h\left(\xi \right):\mathsf{\Gamma }\to {\mathbb{R}}^{N_h}$ defined by

$${\mathbf{f}}_h\left(\xi \right)={\left[f_M\left(x^{\left(1\right)},\xi \right),\cdots ,f_M\left(x^{\left(N_h\right)},\xi \right)\right]}^T\in {\mathbb{R}}^{N_h}$$

for $\xi \in \mathsf{\Gamma }$, where $f_M(x,\xi )=exp\left(z_M(x,\xi )\right)$.

Figure 6a,b show the error for SDEIM with dimensions $M=33$ and $M=109$ for $\tilde{N}={10}^3$ samples respectively. The tolerance is set to $tol={10}^{-3}$, the threshold is set to $\epsilon =0.30$ and the confidence is set to $0.99$ for $\delta ={10}^{-2}$ in Algorithm 2. Here the black points are the samples satisfying $\eta \left(\xi \right)\le tol$ and the red crosses are the failed samples satisfying $\eta \left(\xi \right)>tol$. The empirical probabilities $\overline{p}$ are $\overline{p}=0.062$ and $\overline{p}=0.051$, which are both smaller than our probability threshold $\epsilon =0.30$.

Figure 6. (a) The error for SDEIM with $M=33$. (b) The error for SDEIM with $M=109$. Each point or fork corresponds to a sample of the parameter.

Table 3 shows more details about the property of SDEIM in different parameter settings. The data in Table 3 are calculated with $\tilde{N}={10}^4$ samples and the data in Figure 6 are the first ${10}^3$ of these ${10}^4$ samples. From Table 3, it can be seen that for different dimensions of parameter spaces ($M=33$ and $M=109$) and different tolerances ($tol={10}^{-1},{10}^{-2},{10}^{-3}$), SDEIM can produce a linear approximation space $\mathcal{V}$ satisfying different probability thresholds ($\epsilon =0.70,0.50,0.30,0.10$). Table 4 shows the average number of comparisons for each basis function in SDEIM with different parameter settings. The average number of comparisons is the ratio of the number of comparisons to the number of basis functions for the linear approximation space $\mathcal{V}$. It is clear that SDEIM has a very small number of comparisons to generate each basis function.

Table 3. The empirical probability $\overline{p}$ for different parameter settings in SDEIM.

		$\mathit{\epsilon }=0.70$	$\mathit{\epsilon }=0.50$	$\mathit{\epsilon }=0.30$	$\mathit{\epsilon }=0.10$
		$\mathit{N}=\mathbf{5}$	$\mathit{N}=\mathbf{10}$	$\mathit{N}=\mathbf{26}$	$\mathit{N}=\mathbf{231}$
$M=33$	$tol={10}^{-1}$	$3.060\times {10}^{-1}$	$2.189\times {10}^{-1}$	$4.770\times {10}^{-2}$	$2.700\times {10}^{-3}$
	$tol={10}^{-2}$	$2.766\times {10}^{-1}$	$1.550\times {10}^{-1}$	$6.420\times {10}^{-2}$	$6.700\times {10}^{-3}$
	$tol={10}^{-3}$	$5.344\times {10}^{-1}$	$2.406\times {10}^{-1}$	$5.550\times {10}^{-2}$	$1.890\times {10}^{-2}$
$M=109$	$tol={10}^{-1}$	$5.664\times {10}^{-1}$	$2.567\times {10}^{-1}$	$1.173\times {10}^{-1}$	$1.050\times {10}^{-2}$
	$tol={10}^{-2}$	$4.233\times {10}^{-1}$	$2.143\times {10}^{-1}$	$4.740\times {10}^{-2}$	$9.900\times {10}^{-3}$
	$tol={10}^{-3}$	$1.863\times {10}^{-1}$	$1.403\times {10}^{-1}$	$5.020\times {10}^{-2}$	$4.400\times {10}^{-3}$

Table 4. The average number of comparisons for each basis function in SDEIM with different parameter settings.

		$\mathit{\epsilon }=0.70$	$\mathit{\epsilon }=0.50$	$\mathit{\epsilon }=0.30$	$\mathit{\epsilon }=0.10$
$M=33$	$tol={10}^{-1}$	$1.1037$	$1.1841$	$1.7954$	$5.4669$
	$tol={10}^{-2}$	$1.0612$	$1.1284$	$1.3482$	$2.8485$
	$tol={10}^{-3}$	$1.0346$	$1.1065$	$1.5488$	$2.5069$
$M=109$	$tol={10}^{-1}$	$1.0372$	$1.1270$	$1.2069$	$2.6813$
	$tol={10}^{-2}$	$1.0172$	$1.0527$	$1.1066$	$1.8194$
	$tol={10}^{-3}$	$1.0291$	$1.0446$	$1.1196$	$2.2821$

4.4. The FitzHugh–Nagumo (F-N) System

This test problem considers the F-N system, which is a simplified model of spiking neuron activation and deactivation dynamics [6,16]. Within the F-N system, the nonlinear function $f\left(v\right)$ is defined as

$$f\left(v\right)=v(v-0.01)(1-v),$$

(28)

where v is the voltage and satisfies the F-N system

$$\begin{array}{cc}\hfill \epsilon v_t(x,t)& ={\epsilon }^2{v}_{xx}(x,t)+f\left(v(x,t)\right)-w(x,t)+c,\hfill \\ \hfill w_t(x,t)& =bv(x,t)-\gamma w(x,t)+c,\hfill \end{array}$$

(29)

with $\epsilon =0.015,b=0.5,\gamma =2$, $c=0.05$ and $x\in \mathsf{\Omega }=[0,L],t\ge 0$. The variable w represents the recovery of voltage, and the variable L is set to $L=1$. Following the settings in [6], initial and boundary conditions are set to

$$\begin{array}{cccccc}\hfill v(x,0)& =0,\hfill & \hfill w(x,0)& =0,\hfill & \hfill x& \in \mathsf{\Omega }=[0,L],\hfill \\ \hfill v_x(0,t)& =-i_0\left(t\right),\hfill & \hfill v_x(L,t)& =0,\hfill & \hfill t& \ge 0,\hfill \end{array}$$

(30)

where the stimulus $i_0\left(t\right)=50,000t^{3}exp(-15t)$. We discretize the physical domain $\mathsf{\Omega }$ using a uniform grid with $N_h=2500$. The dimension of the finite difference system is 2500. We take 2501 time nodes at evenly in the interval $[0,8]$, of which 500 are randomly selected as the candidate sample set ${\mathsf{\Gamma }}_{\mathrm{train}}$ for DEIM and SDEIM training, and the remainder are utilized to test the probability features.

The solution of the F-N system has a limit cycle for each spatial variable x, and we display the phase-space diagram of v and w at various spatial positions in Figure 7. Symmetry in the limit cycles of the F-N system is well captured by SDEIM, as seen in Figure 7. Tolerances are set to $tol={10}^{-3}$ for solutions $v,w$ and $tol={10}^{-4}$ for nonlinear function $f\left(v\right)$. The probability of errors exceeding the corresponding tolerances among 1500 verifying samples are depicted in Figure 8a. The probabilities of v converge to a constant value in this case, because the accuracy of the solution v is dependent on the accuracy of the solution w for both DEIM and SDEIM. Figure 8b shows the number of comparisons increases as the number of basis functions increases. The black dotted line represents the number of comparisons in DEIM, which is $\left(2\right|{\mathsf{\Gamma }}_{\mathrm{train}}|-n+1)n/2$ (n is the number of the basis functions). It is clear that SDEIM is very efficient solve this F-N system.

Figure 7. (a) Phase-space diagram of v and w at different spatial points x from the FD system ($dim2500$), DEIM reduced system ($dim67$) and SDEIM reduced system ($dim72$). (b) Corresponding projection of the solution onto the $v-w$ plane.

Figure 8. (a) Probability properties for DEIM reduced system and SDEIM reduced system. (b) The number of comparisons for DEIM reduced system and SDEIM reduced system.

5. Conclusions

Empirical interpolation is a widely used model reduction method for nonlinear and nonaffine parameterized systems through a separate representation of the spatial and the parametric variables. This kind of method typically requires a fine candidate sample set, which can lead to high computational costs. With a focus on randomized computational methods, we in this paper propose a stochastic discrete empirical interpolation method (SDEIM). In SDEIM, large candidate sample sets are replaced by gradually generated random samples, such that the computational costs of constructing the corresponding interpolation formulation can be dramatically reduced. With our analysis, the stopping criterion based on a probability threshold in SDEIM can guarantee the interpolation is accurate with a given confidence. Our numerical results show that this randomized approach is efficient, especially for variable separation for high dimensional nonlinear and nonaffine systems. However, as we use the Hoeffding’s inequality to estimate the failure probability in the verifying stage, SDEIM is efficient when this probability is not too small, but it can be inefficient when the probability is very small. For applying SDEIM to systems which require high reliability, current efforts are focused on combining subset simulation methods, and implementing such strategies will be the focus of our future work.