Parameter Estimation for Several Types of Linear Partial Differential Equations Based on Gaussian Processes

Abstract

This paper mainly considers the parameter estimation problem for several types of differential equations controlled by linear operators, which may be partial differential, integro-differential and fractional order operators. Under the idea of data-driven methods, the algorithms based on Gaussian processes are constructed to solve the inverse problem, where we encode the distribution information of the data into the kernels and construct an efficient data learning machine. We then estimate the unknown parameters of the partial differential Equations (PDEs), which include high-order partial differential equations, partial integro-differential equations, fractional partial differential equations and a system of partial differential equations. Finally, several numerical tests are provided. The results of the numerical experiments prove that the data-driven methods based on Gaussian processes not only estimate the parameters of the considered PDEs with high accuracy but also approximate the latent solutions and the inhomogeneous terms of the PDEs simultaneously.

1. Introduction

In the era of big data, the study of data-driven methods and probabilistic machine-learning methods has increasingly attracted researchers [1,2,3]. Exploiting data-driven methods and machine-learning methods to solve the forward and inverse problem of partial differential equations (PDEs) has been valued [4,5,6,7,8,9,10,11], and (deep) neural networks are preferred [12,13,14,15,16,17,18,19,20].

Compared with other machine-learning methods, such as regularized least-squares classifiers (RLSCs) [21] and support vector machines (SVMs) [22], Gaussian processes possess a strict mathematical basis and coincide with the Bayesian estimation method in essence [23]. Raissi et al. (2017) [24] considered a new algorithm for the inverse problem of the PDEs controlled by linear operators based on Gaussian process regression. Different from classical methods, such as the Tikhonov regularization method [25], Gaussian processes solve inverse problems of the PDEs from the perspective of statistical inference.

In the Gaussian processes, the Bayesian method is introduced to encode the distribution information of the data into structured prior information [26], so as to construct an efficient data learning machine, estimate the unknown parameters of the PDEs, infer the solution of the considered equations and quantify the uncertainty of the prediction solution. The research on Gaussian processes solving the forward and inverse problem of the PDEs is a new branch of probabilistic numerics in numerical analysis [27,28,29,30,31,32,33,34].

In this paper, the inverse problem is to estimate the unknown parameters of the considered equations from (noisy) observation data. We extend the Gaussian processes method to deal with the inverse problem of several types of PDEs, which include high-order partial differential equations, partial integro-differential equations, fractional partial differential equations and the system of partial differential equations.

Finally, several numerical tests are provided. The results of the numerical experiments prove that the data-driven methods based on Gaussian processes not only estimate the parameters of the considered PDEs with high accuracy but also approximate the latent solutions and the inhomogeneous terms of the PDEs simultaneously.

The sections of this paper are organized as follows. In Section 2, the basic workflow of Gaussian processes solving the inverse problems of the PDEs is provided. Section 3 describes the estimation algorithm of the fractional PDEs and the linear PDEs system based on Gaussian processes in detail. In Section 4, numerical experiments are performed to prove the validity of the proposed methodology. Finally, our conclusions are given in Section 5.

2. Mathematical Model and Methodology

The following partial differential equations are considered in this paper,

$$\begin{array}{c}\hfill {\mathcal{L}}_x^{\varphi }u\left(x\right)=f\left(x\right),\end{array}$$

(1)

where x represents a vector of D dimensions, $u\left(x\right)$ is the latent solution of (1), $f\left(x\right)$ is the inhomogeneous term, ${\mathcal{L}}_x^{\varphi }$ is a linear operator, and $\varphi$ represents the unknown parameter. For the sake of simplicity, we can introduce the following heat equation as an example,

$$\begin{array}{c}\hfill {\mathcal{L}}_{(t,x)}^{\varphi }u(t,x):=\frac{\partial }{\partial t}u(t,x)-\alpha \frac{{\partial }^2}{\partial x^2}u(t,x)=f(t,x),\end{array}$$

(2)

where heat diffusivity $\alpha$ is the unknown parameter $\varphi$.

Assume that we obtain observations $\left\{x_u,y_u\right\}$ and $\left\{x_f,y_f\right\}$ from the latent solution $u\left(x\right)$ and the inhomogeneous term $f\left(x\right)$, respectively, then we can estimate the unknown parameter $\varphi$ and approximate the latent solutions of the forward differential equations system according to the posterior results. However, one of the advantages of the methods used in this paper is that we do not need to consider the initial and boundary conditions of problems, because (noisy) observation data from the latent solution and the inhomogeneous term can give enough distribution information of the functions.

As with other machine-learning methods, Gaussian processes can be applied to solve regression problem and classification problem. Moreover Gaussian processes can be seen as a class of methods called kernel machines [35]. However, compared with other kernel machine methods, such as support vector machines (SVMs) and relevance vector machines (RVMs) [36], the strict probability theory of Gaussian processes limits the popularization of this method in industrial circles. Another drawback of Gaussian processes is that the computational cost may be expensive.

2.1. Gaussian Process Prior

Take a Gaussian process prior hypothesis as follows,

$$\begin{array}{c}\hfill u\left(x\right)\sim \mathcal{G}\mathcal{P}(0,k_{uu}(x,x^{\prime };\theta )),\end{array}$$

(3)

and assume that covariance function $k_{uu}(x,x^{\prime };\theta )$ has the following squared exponential form,

$$\begin{array}{c}\hfill k_{uu}(x,x^{\prime };\theta )={\sigma }_u^{2}exp(-\frac{1}{2}\sum _{d=1}^D{w}_d{(x_d-{x^{\prime }}_d)}^2),\end{array}$$

(4)

where $\theta$ is the hyper-parameters of the kernel (covariance function) $k_{uu}$, for Equation (4), $\theta =\left({\sigma }_u^2,{\left(w_d\right)}_{d=1}^D\right)$. Any prior information of $u\left(x\right)$, such as monotonicity and periodicity, can be encoded into the kernel $k_{uu}$.

Since the linear transformation of Gaussian processes, such as differentiation and integration, is still Gaussian, we can obtain

$$\begin{array}{c}\hfill {\mathcal{L}}_x^{\varphi }u\left(x\right)=f\left(x\right)\sim \mathcal{G}\mathcal{P}(0,k_{ff}(x,x^{\prime };\theta ,\varphi )),\end{array}$$

(5)

where the covariance function is $k_{ff}(x,x^{\prime };\theta ,\varphi )={\mathcal{L}}_x^{\varphi }{\mathcal{L}}_{x^{\prime }}^{\varphi }{k}_{uu}(x,x^{\prime };\theta )$.

Moreover, the covariance function between $u\left(x\right)$ and $f\left(x^{\prime }\right)$ is $k_{uf}(x,x^{\prime };\theta ,\varphi )=$${\mathcal{L}}_{x^{\prime }}^{\varphi }{k}_{uu}(x,x^{\prime };\theta )$ and the covariance function between $f\left(x\right)$ and $u\left(x^{\prime }\right)$ is $k_{fu}(x,x^{\prime };\theta ,\varphi )={\mathcal{L}}_x^{\varphi }{k}_{uu}(x,x^{\prime };\theta )$. The proposal and reasoning of Equation (5) are the core of Gaussian processes to estimate the unknown parameters of the PDEs [24]. In this step, the parameter information of the PDEs is encoded into the kernels $k_{ff}$, $k_{uf}$ and $k_{fu}$. Furthermore, we can utilize the joint density function of $u\left(\mathit{x}\right)$ and $f\left(\mathit{x}\right)$ for maximum likelihood estimation of parameters $\varphi$. The greatest contribution of Gaussian process regression is that we transform the unknown parameters $\varphi$ of the linear operator ${\mathcal{L}}_x^{\varphi }$ into the hyper-parameters of the kernels $k_{ff}$, $k_{uf}$ and $k_{fu}$.

By Mercer’s theorem [37], a positive definite covariance function $k(x,x^{\prime })$ can be decomposed into $k(x,x^{\prime })={\displaystyle \sum _{i=1}^{\infty }}{\lambda }_i{\phi }_i\left(x\right){\phi }_i\left(x^{\prime }\right)$, where ${\lambda }_i$ and ${\phi }_i$ are eigenvalues and eigenfunctions, respectively, satisfying $\int k(x,x^{\prime }){\phi }_i\left(x\right)dx={\lambda }_i{\phi }_i\left(x^{\prime }\right)$. ${\left\{\sqrt{{\lambda }_i}{\phi }_i\right\}}_{i=1}^{\infty }$ is treated as a set of orthogonal basis and a reproducing kernel Hilbert space (RKHS) is constructed [38], which can be seen as a kernel trick [38]. However, the covariance function in Equation (4) has no finite decomposition. Different covariance functions, such as rational quadratic covariance functions and Matérn covariance functions should be selected under different prior information [23]. The most important thing is that the kernel considered should cover the prior information [24].

2.2. Data Training

From the properties of the Gaussian processes [23], we find

$$\begin{array}{c}\hfill \mathit{y}\sim \mathcal{G}\mathcal{P}(\mathbf{0},\mathit{K}),\end{array}$$

(6)

where $\mathit{y}=\left[\begin{array}{c}{\mathit{y}}_u\\ {\mathit{y}}_f\end{array}\right]$, $\mathit{K}=\left[\begin{array}{cc}{k}_{uu}({\mathit{x}}_u,{\mathit{x}}_u;\theta )+{\sigma }_u^{2}I& k_{uf}({\mathit{x}}_u,{\mathit{x}}_f;\theta ,\varphi )\\ k_{fu}({\mathit{x}}_f,{\mathit{x}}_u;\theta ,\varphi )& k_{ff}({\mathit{x}}_f,{\mathit{x}}_f;\theta ,\varphi )+{\sigma }_f^{2}I\end{array}\right]$.

According to (6), we train the parameters $\varphi$ and the hyper-parameters $\theta$ by minimizing the following negative log marginal likelihood,

$$\begin{array}{c}\hfill -logp\left(\mathit{y}\right|{\mathit{x}}_u,{\mathit{x}}_f,\varphi ,\theta ,{\sigma }_{n_u}^2,{\sigma }_{n_f}^2)=\frac{1}{2}log\left|\mathit{K}\right|+\frac{1}{2}{\mathit{y}}^T{\mathit{K}}^{-1}\mathit{y}+\frac{N}{2}log2\pi ,\end{array}$$

(7)

where N is the length of $\mathit{y}$, and we consider a Quasi–Newton optimizer $L-BFGS-B$ for training [39].

Add noise on the observed data of formula (6), we introduce notations that ${\mathit{y}}_u=u\left({\mathit{x}}_u\right)+{\epsilon }_u$ and ${\mathit{y}}_f=u\left({\mathit{x}}_f\right)+{\epsilon }_f$, where ${\epsilon }_u\sim N(\mathbf{0},{\sigma }_u^{2}I)$ and ${\epsilon }_f\sim N(\mathbf{0},{\sigma }_f^{2}I)$. Assume that ${\epsilon }_u$ and ${\epsilon }_f$ are mutually independent additive noise.

The training procedure is the core of the algorithm, which reveals the “regression nature” of Gaussian processes. Furthermore, it is worth mentioning that the negative log marginal likelihood (7) is not only suitable for training the model, it automatically trades off between data-fit and model complexity. While minimizing the term $\frac{1}{2}{y}^T{K}^{-1}y$ in Equation (7), we use the term $log\left|K\right|$ to penalize the model complexity [23]. This regularization-like mechanism is a key property of Gaussian process regression, which effectively prevents overfitting.

In truth, model training is solving parameters $\varphi$ and hyper-parameters $\{\theta ,{\sigma }_u^2,{\sigma }_f^2\}$ by minimizing Equation (7), and this defines a non-convex optimization problem, which is friendly to machine learning. Heuristic algorithms can be introduced to solve it, such as the whale optimization algorithm [40] and the ant colony optimization algorithm [41]. It is also worth noting that the computational cost of training has a cubic relationship with the amount of the training data due to the Cholesky decomposition of the covariance functions in (7), and the papers [42,43,44] considered this conundrum.

2.3. Gaussian Process Posterior

By the conditional distribution of Gaussian processes [23], the posterior distribution of the prediction $u\left(x^{\ast }\right)$ at point $x^{\ast }$ can be directly written as

$$\begin{array}{c}\hfill u\left(x^{\ast }\right)\left|\left[\begin{array}{c}{\mathit{y}}_u\hfill \\ {\mathit{y}}_f\hfill \end{array}\right]\right.\sim \mathcal{G}\mathcal{P}({\mathit{q}}_u^T{\mathit{K}}^{-1}\mathit{y},k_{uu}(x^{\ast },x^{\ast })-{\mathit{q}}_u^T{\mathit{K}}^{-1}{\mathit{q}}_u),\end{array}$$

(8)

where ${\mathit{q}}_u^T=\left[k_{uu}(x^{\ast },{\mathit{x}}_u),k_{uf}(x^{\ast },{\mathit{x}}_f)\right]$.

By the deduction of [11], the posterior distribution of the prediction $f\left(x^{\ast }\right)$ at the point $x^{\ast }$ can be written as

$$\begin{array}{c}\hfill f\left(x^{\ast }\right)\left|\left[\begin{array}{c}{\mathit{y}}_u\hfill \\ {\mathit{y}}_f\hfill \end{array}\right]\right.\sim \mathcal{G}\mathcal{P}({\mathit{q}}_f^T{\mathit{K}}^{-1}\mathit{y},k_{ff}(x^{\ast },x^{\ast })-{\mathit{q}}_f^T{\mathit{K}}^{-1}{\mathit{q}}_f),\end{array}$$

(9)

where ${\mathit{q}}_f^T=\left[k_{fu}(x^{\ast },{\mathit{x}}_u),k_{ff}(x^{\ast },{\mathit{x}}_f)\right]$.

The posterior mean of (8) and (9) can be seen as the predicted solution of $u\left(x\right)$ and $f\left(x\right)$, respectively. Furthermore, the posterior variance is the direct result of the Bayesian method, which can be used to measure the reliability of the prediction solution.

3. Inverse Problem for Fractional PDEs and the System of Linear PDEs

This section provides the process to estimate the unknown parameters of fractional PDEs and the system of linear PDEs in detail when Gaussian processes are used. However, the processing of these two types of PDEs is more complicated than the other two types of PDEs considered in this paper.

3.1. Processing of Fractional PDEs

The following fractional partial differential equations are considered,

$$\begin{array}{c}\hfill {\mathcal{L}}_{(t,x)}^{\varphi }u(t,x):=\frac{{\partial }^{\alpha }}{\partial t^{\alpha }}u(t,x)-Q\frac{{\partial }^2}{\partial x^2}u(t,x)=f(t,x),\end{array}$$

(10)

where $\varphi =Q$ and the fractional order $\alpha$ is a given parameter, $u(t,x)$ is a continuous real function absolutely integrable in ${\mathbb{R}}^2$, and its arbitrary-order partial derivatives are also continuous function absolutely integrable in ${\mathbb{R}}^2$. The fractional partial derivative in (10) is defined in the Caputo sense [45] as the following

$$\begin{array}{c}\hfill \frac{{\partial }^{\alpha }}{\partial t^{\alpha }}u(t,x)=\frac{1}{\Gamma (n-\alpha )}{\int }_{-\infty }^t\frac{1}{{(t-\tau )}^{\alpha -n+1}}\frac{{\partial }^{n}u(\tau ,x)}{\partial {\tau }^n}d\tau ,\end{array}$$

(11)

where $\alpha$ is a positive number ($n-1<\alpha \le n$, $n\in N_{+}$), and $\Gamma \left(\right)$ is the gamma function.

The key step of solving (10) is the deriving of kernels with the fractional order operators. By [46,47], the four-dimensional Fourier transforms are applied with respect to the temproral variable $(t,t^{\prime })$ and with respect to the spatial variable $(x,x^{\prime })$ on $k_{uu}[(t,x),(t^{\prime },x^{\prime })]$ to obtain ${\widehat{k}}_{uu}[(\upsilon ,\omega ),({\upsilon }^{\prime },{\omega }^{\prime })]$, as the excellent properties of the squared exponential covariance function $k_{uu}[(t,x),(t^{\prime },x^{\prime })]$ assures the feasibility of the Fourier transform of derivatives of $k_{uu}$, on the basis of that $k_{uu}$ and its partial derivatives vanish for $\sqrt{t^2+x^2+{t^{\prime }}^2+{x^{\prime }}^2}\to +\infty$. We find the following intermediate kernels,

$$\begin{array}{c}\hfill \begin{array}{c}{\widehat{k}}_{ff}[(\upsilon ,\omega ),({\upsilon }^{\prime },{\omega }^{\prime })]=[{(-i\upsilon )}^{\alpha }{(-i{\upsilon }^{\prime })}^{\alpha }-Q{(-i\upsilon )}^{\alpha }{(-i{\omega }^{\prime })}^2-\hfill \\ Q{(-i\omega )}^2{(-i{\upsilon }^{\prime })}^{\alpha }+Q^2{(-i\omega )}^2{(-i{\omega }^{\prime })}^2]{\widehat{k}}_{uu}[(\upsilon ,\omega ),({\upsilon }^{\prime },{\omega }^{\prime })],\hfill \\ {\widehat{k}}_{uf}[(\upsilon ,\omega ),({\upsilon }^{\prime },{\omega }^{\prime })]=[{(-i{\upsilon }^{\prime })}^{\alpha }-Q{(-i{\omega }^{\prime })}^2]{\widehat{k}}_{uu}[(\upsilon ,\omega ),({\upsilon }^{\prime },{\omega }^{\prime })],\hfill \\ {\widehat{k}}_{fu}[(\upsilon ,\omega ),({\upsilon }^{\prime },{\omega }^{\prime })]=[{(-i\upsilon )}^{\alpha }-Q{(-i\omega )}^2]{\widehat{k}}_{uu}[(\upsilon ,\omega ),({\upsilon }^{\prime },{\omega }^{\prime })].\hfill \end{array}\end{array}$$

(12)

Then, we perform the inverse Fourier transform on ${\widehat{k}}_{ff}[(\upsilon ,\omega ),({\upsilon }^{\prime },{\omega }^{\prime })]$, ${\widehat{k}}_{uf}[(\upsilon ,\omega ),({\upsilon }^{\prime },{\omega }^{\prime })]$ and ${\widehat{k}}_{fu}[(\upsilon ,\omega ),({\upsilon }^{\prime },{\omega }^{\prime })]$ to obtain kernels $k_{ff}[(t,x),(t^{\prime },x^{\prime })]$, $k_{uf}[(t,x),(t^{\prime },x^{\prime })]$ and $k_{fu}[(t,x),(t^{\prime },x^{\prime })]$, respectively. Furthermore, the other steps are exactly the same as described in Section 2.

3.2. Processing of the System of Linear PDEs

Consider the systems of linear partial differential equations as follows,

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\mathcal{L}}_x^{\varphi ,1}{u}_1\left(x\right)+{\mathcal{L}}_x^{\varphi ,2}{u}_2\left(x\right)=f_1\left(x\right),\\ {\mathcal{L}}_x^{\varphi ,3}{u}_1\left(x\right)+{\mathcal{L}}_x^{\varphi ,4}{u}_2\left(x\right)=f_2\left(x\right),\end{array}\right.\end{array}$$

(13)

where x represents a vector of D dimensions, ${\mathcal{L}}_x^{\varphi ,1}$, ${\mathcal{L}}_x^{\varphi ,2}$, ${\mathcal{L}}_x^{\varphi ,3}$ and ${\mathcal{L}}_x^{\varphi ,4}$ are linear operators, and $\varphi$ denotes the unkonwn parameters of (13).

Assume the following prior hypotheses

$$\begin{array}{c}\hfill \begin{array}{c}{u}_1\left(x\right)\sim \mathcal{G}\mathcal{P}(0,k_{uu}^{11}(x,x^{\prime };{\theta }_1)),\hfill \\ u_2\left(x\right)\sim \mathcal{G}\mathcal{P}(0,k_{uu}^{22}(x,x^{\prime };{\theta }_2)),\hfill \end{array}\end{array}$$

(14)

to be two mutually independent Gaussian processes, and the covariance functions have a squared exponential form (4). Adding noise to the observed data, we introduce notations that ${\mathit{y}}_{u_1}=u_1\left({\mathit{x}}_{u_1}\right)+{\epsilon }_{u_1}$, ${\mathit{y}}_{u_2}=u_2\left({\mathit{x}}_{u_2}\right)+{\epsilon }_{u_2}$, ${\mathit{y}}_{f_1}=f_1\left({\mathit{x}}_{f_1}\right)+{\epsilon }_{f_1}$ and ${\mathit{y}}_{f_2}=f_2\left({\mathit{x}}_{f_2}\right)+{\epsilon }_{f_2}$, where ${\epsilon }_{u_1}\sim N(\mathbf{0},{\sigma }_{u_1}^{2}I)$, ${\epsilon }_{u_2}\sim N(\mathbf{0},{\sigma }_{u_2}^{2}I)$, ${\epsilon }_{f_1}\sim N(\mathbf{0},{\sigma }_{f_1}^{2}I)$ and ${\epsilon }_{f_2}\sim N(\mathbf{0},{\sigma }_{f_2}^{2}I)$.

According to the prior hypotheses, we find

$$\begin{array}{c}\hfill \mathit{y}\sim \mathcal{G}\mathcal{P}(\mathbf{0},\mathit{K}),\end{array}$$

(15)

where $\mathit{y}=\left[\begin{array}{c}{\mathit{y}}_{u_1}\\ {\mathit{y}}_{u_2}\\ {\mathit{y}}_{f_1}\\ {\mathit{y}}_{f_2}\end{array}\right]$, $\mathit{K}=\left[\begin{array}{cccc}{k}_{uu}^{11}+{\sigma }_{n_{u_1}}^2{I}_{n_{u_1}}& k_{uu}^{12}& k_{uf}^{11}& k_{uf}^{12}\\ k_{uu}^{21}& k_{uu}^{22}+{\sigma }_{n_{u_2}}^2{I}_{n_{u_2}}& k_{uf}^{21}& k_{uf}^{22}\\ k_{fu}^{11}& k_{fu}^{12}& k_{ff}^{11}+{\sigma }_{n_{f_1}}^2{I}_{n_{f_1}}& k_{ff}^{12}\\ k_{fu}^{21}& k_{fu}^{22}& k_{ff}^{21}& k_{ff}^{22}+{\sigma }_{n_{f_2}}^2{I}_{n_{f_2}}\end{array}\right],$ $k_{uu}^{12}=0,k_{uf}^{11}={\mathcal{L}}_{x^{\prime }}^{\varphi ,1}{k}_{uu}^{11},k_{uf}^{12}={\mathcal{L}}_{x^{\prime }}^{\varphi ,3}{k}_{uu}^{11},k_{uf}^{21}={\mathcal{L}}_{x^{\prime }}^{\varphi ,2}{k}_{uu}^{22},k_{uf}^{22}={\mathcal{L}}_{x^{\prime }}^{\varphi ,4}{k}_{uu}^{22},k_{uu}^{21}=0,$$k_{fu}^{11}={\mathcal{L}}_x^{\varphi ,1}{k}_{uu}^{11},k_{fu}^{12}={\mathcal{L}}_x^{\varphi ,2}{k}_{uu}^{22},k_{fu}^{21}={\mathcal{L}}_x^{\varphi ,3}{k}_{uu}^{11},k_{fu}^{22}={\mathcal{L}}_x^{\varphi ,4}{k}_{uu}^{22},k_{ff}^{11}={\mathcal{L}}_x^{\varphi ,1}{\mathcal{L}}_{x^{\prime }}^{\varphi ,1}{k}_{uu}^{11}+$${\mathcal{L}}_x^{\varphi ,2}{\mathcal{L}}_{x^{\prime }}^{\varphi ,2}{k}_{uu}^{22},k_{ff}^{12}={\mathcal{L}}_x^{\varphi ,1}{\mathcal{L}}_{x^{\prime }}^{\varphi ,3}{k}_{uu}^{11}+{\mathcal{L}}_x^{\varphi ,2}{\mathcal{L}}_{x^{\prime }}^{\varphi ,4}{k}_{uu}^{22},k_{ff}^{21}={\mathcal{L}}_x^{\varphi ,3}{\mathcal{L}}_{x^{\prime }}^{\varphi ,1}{k}_{uu}^{11}+{\mathcal{L}}_x^{\varphi ,4}{\mathcal{L}}_{x^{\prime }}^{\varphi ,2}{k}_{uu}^{22},k_{ff}^{22}={\mathcal{L}}_x^{\varphi ,3}{\mathcal{L}}_{x^{\prime }}^{\varphi ,3}{k}_{uu}^{11}+{\mathcal{L}}_x^{\varphi ,4}{\mathcal{L}}_{x^{\prime }}^{\varphi ,4}{k}_{uu}^{22}.$

According to (15), parameters $\varphi$ and hyper-parameters $\{{\theta }_1,{\theta }_2\}$ can be trained by minimizing the following negative log marginal likelihood

$$-logp\left(\mathit{y}\right|{\mathit{x}}_{u_1},{\mathit{x}}_{u_2},{\mathit{x}}_{f_1},{\mathit{x}}_{f_2},\varphi ,{\theta }_1,{\theta }_2,{\sigma }_{n_{u_1}}^2,{\sigma }_{n_{u_2}}^2,{\sigma }_{n_{f_1}}^2,{\sigma }_{n_{f_2}}^2)=\frac{1}{2}log\left|\mathit{K}\right|+\frac{1}{2}{\mathit{y}}^T{\mathit{K}}^{-1}\mathit{y}+\frac{N}{2}log2\pi ,$$

where N is the length of $\mathit{y}$.

After the training step, we can write the posterior distribution of $u_1$, $u_2$, $f_1$ and $f_2$ as follows,

$$\begin{array}{c}\hfill \begin{array}{c}{u}_1\left(x^{\ast }\right)\left|\mathit{y}\right.\sim \mathcal{G}\mathcal{P}({\mathit{q}}_{u_1}^T{\mathit{K}}^{-1}\mathit{y},k_{uu}^{11}(x^{\ast },x^{\ast })-{\mathit{q}}_{u_1}^T{\mathit{K}}^{-1}{\mathit{q}}_{u_1}),\hfill \\ u_2\left(x^{\ast }\right)\left|\mathit{y}\right.\sim \mathcal{G}\mathcal{P}({\mathit{q}}_{u_2}^T{\mathit{K}}^{-1}\mathit{y},k_{uu}^{22}(x^{\ast },x^{\ast })-{\mathit{q}}_{u_2}^T{\mathit{K}}^{-1}{\mathit{q}}_{u_2}),\hfill \\ f_1\left(x^{\ast }\right)\left|\mathit{y}\right.\sim \mathcal{G}\mathcal{P}({\mathit{q}}_{f_1}^T{\mathit{K}}^{-1}\mathit{y},k_{ff}^{11}(x^{\ast },x^{\ast })-{\mathit{q}}_{f_1}^T{\mathit{K}}^{-1}{\mathit{q}}_{f_1}),\hfill \\ f_2\left(x^{\ast }\right)\left|\mathit{y}\right.\sim \mathcal{G}\mathcal{P}({\mathit{q}}_{f_2}^T{\mathit{K}}^{-1}\mathit{y},k_{ff}^{22}(x^{\ast },x^{\ast })-{\mathit{q}}_{f_2}^T{\mathit{K}}^{-1}{\mathit{q}}_{f_2}),\hfill \end{array}\end{array}$$

(16)

where

$$\begin{array}{c}\hfill \begin{array}{c}{\mathit{q}}_{u_1}^T=\left[k_{uu}^{11}(x^{\ast },{\mathit{x}}_{u_1}),k_{uu}^{12}(x^{\ast },{\mathit{x}}_{u_2}),k_{uf}^{11}(x^{\ast },{\mathit{x}}_{f_1}),k_{uf}^{12}(x^{\ast },{\mathit{x}}_{f_2})\right],\hfill \\ {\mathit{q}}_{u_2}^T=\left[k_{uu}^{21}(x^{\ast },{\mathit{x}}_{u_1}),k_{uu}^{22}(x^{\ast },{\mathit{x}}_{u_2}),k_{uf}^{21}(x^{\ast },{\mathit{x}}_{f_1}),k_{uf}^{22}(x^{\ast },{\mathit{x}}_{f_2})\right],\hfill \\ {\mathit{q}}_{f_1}^T=\left[k_{fu}^{11}(x^{\ast },{\mathit{x}}_{u_1}),k_{fu}^{12}(x^{\ast },{\mathit{x}}_{u_2}),k_{ff}^{11}(x^{\ast },{\mathit{x}}_{f_1}),k_{ff}^{12}(x^{\ast },{\mathit{x}}_{f_2})\right],\hfill \\ {\mathit{q}}_{f_2}^T=\left[k_{fu}^{21}(x^{\ast },{\mathit{x}}_{u_1}),k_{fu}^{22}(x^{\ast },{\mathit{x}}_{u_2}),k_{ff}^{21}(x^{\ast },{\mathit{x}}_{f_1}),k_{ff}^{22}(x^{\ast },{\mathit{x}}_{f_2})\right].\hfill \end{array}\end{array}$$

(17)

4. Numerical Tests

This section provides four examples to prove the validity of the methodology proposed. We consider the inverse problem of four types of PDEs, which include high-order partial differential equations, partial integro-differential equations, fractional partial differential equations and the system of partial differential equations.

Introduce the relative error ${\mathcal{L}}^2$ between the exact solution and the prediction solution to represent the prediction error of the algorithm,

$$\begin{array}{c}\hfill {\mathcal{L}}^2=\sqrt{\sum _{x^{\ast }}{[u\left(x^{\ast }\right)-\overline{u}\left(x^{\ast }\right)]}^2}/\phantom{\sqrt{\sum _{x^{\ast }}{[u\left(x^{\ast }\right)-\overline{u}\left(x^{\ast }\right)]}^2}\sqrt{\sum _{x^{\ast }}{\left[u\left(x^{\ast }\right)\right]}^2}}\sqrt{\sum _{x^{\ast }}{\left[u\left(x^{\ast }\right)\right]}^2},\end{array}$$

(18)

where $x^{\ast }$ represents the predicted point, $u\left(x^{\ast }\right)$ is the exact solution at this point, and $\overline{u}\left(x^{\ast }\right)$ is the corresponding prediction solution.

4.1. Simulation for a High-Order Partial Differential Equation

Example 1.

$$\begin{array}{c}\hfill {\mathcal{L}}_{(t,x)}^{\varphi }u(t,x):=u_{tttt}-\alpha u_{ttxx}+\beta u_{xxxxxxx}=f,\end{array}$$

(19)

where $(t,x)\in [0,1]\times [0,2\pi ]$, $\varphi =(\alpha ,\beta )$. The exact solution is $u(t,x)=e^{-t}[cos\left(x\right)+sin\left(x\right)]$ and the inhomogeneous term is $f(t,x)=(-1+\alpha -\beta )e^{-t}[cos\left(x\right)+sin\left(x\right)]$.

Numerical experiments are performed with the noiseless data of Example 1. We denote that the amount of the training data of $u(t,x)$ is $N_u$, and the number of the training data of $f(t,x)$ is $N_f$. Take $N_u=50$ and $N_f=40$ in this subsection. Fix $(\alpha ,\beta )$ to be (1,1), and the estimated value $(\widehat{\alpha },\widehat{\beta })$ is (1.014005, 1.014634). Figure 1 shows the distribution of the training data of $u(t,x)$ and $f(t,x)$. Figure 2 shows the estimation error $|\overline{u}(t,x)-u(t,x)|$ and $|\overline{f}(t,x)-f(t,x)|$ of $u(t,x)$ and $f(t,x)$, respectively. Figure 3 shows the posterior standard deviation of the corresponding prediction solution. It can be seen from Figure 2 and Figure 3 that the posterior standard deviation is positively correlated with the prediction error. The posterior distribution of Gaussian processes can return a satisfactory numerical approximation to $u(t,x)$ and $f(t,x)$.

Table 1. High-order partial differential equation: estimated values of $(\alpha ,\beta )$ and the relative error ${\mathcal{L}}^2$ for $u(t,x)$ and $f(t,x)$, where $N_u=50$, $N_f=40$, and $(\alpha ,\beta )$ is fixed to be (1, 1), (1, 2), (2, 1) and (2, 2) in turn.

$\widehat{\alpha }$	1.014005	1.021724	2.006856	2.008363
$\widehat{\beta }$	1.014634	2.032989	1.006408	2.020304
${\mathcal{L}}^2$ for u	3.201 × 10${}^{-4}$	5.615 × 10${}^{-4}$	1.869 × 10${}^{-4}$	3.319 × 10${}^{-4}$
${\mathcal{L}}^2$ for f	2.665 × 10${}^{-3}$	1.825 × 10${}^{-3}$	3.037 × 10${}^{-3}$	2.823 × 10${}^{-3}$

shows the estimation values of $(\alpha ,\beta )$ and the relative error ${\mathcal{L}}^2$ for $u(t,x)$ and $f(t,x)$, where $(\alpha ,\beta )$ is fixed to be (1, 1), (1, 2), (2, 1) and (2, 2), in turn. Furthermore, the experimental results prove that the results of parameter estimation based on Gaussian processes have relatively high accuracy when high-order PDEs are considered.

4.2. Simulation for a Fractional Partial Differential Equation

Example 2.

$$\begin{array}{c}\hfill {\mathcal{L}}_{(t,x)}^{\varphi }u(t,x):=\frac{{\partial }^{\alpha }}{\partial t^{\alpha }}u(t,x)-Q\frac{{\partial }^2}{\partial x^2}u(t,x)=f(t,x),\end{array}$$

(20)

where $(t,x)\in (-\infty ,1]\times [0,1]$$\varphi =Q$ and the fractional order α is a given parameter. Assume $0<\alpha \le 1$, and the fractional partial derivative in (20) is defined in the Caputo sense as the following,

$$\begin{array}{c}\hfill \frac{{\partial }^{\alpha }}{\partial t^{\alpha }}u(t,x)=\frac{1}{\Gamma (1-\alpha )}{\int }_{-\infty }^t\frac{1}{{(t-\tau )}^{\alpha }}\frac{\partial u(\tau ,x)}{\partial \tau }d\tau ,\end{array}$$

(21)

The exact solution is $u(t,x)=e^t{(1-x)}^2{x}^2$ and the inhomogeneous term is $f(t,x)=\frac{{(1-x)}^2{x}^2}{\Gamma (1-\alpha )}{\int }_{-\infty }^t\frac{e^{\tau }}{{(t-\tau )}^{\alpha }}d\tau -2Q{e}^t[1+6(-1+x)x]$.

Experiments are performed with noiseless data of Example 2.

Table 2. Fractional partial differential equation: estimated values of Q and the relative error ${\mathcal{L}}^2$ for $u(t,x)$ and $f(t,x)$ where $N_u=15$, $N_f=12$, $(\alpha ,Q)$ is fixed to be (0.25, 1), (0.5, 1), (0.75, 1), (0.25, 2), (0.5, 2) and (0.75, 2), in turn.

$(\mathit{\alpha },\mathit{Q})$	$(0.25,1)$	$(0.5,1)$	$(0.75,1)$	$(0.25,2)$	$(0.5,2)$	$(0.75,2)$
$\widehat{Q}$	1.000262	1.000115	0.999436	2.000810	1.999836	1.995571
${\mathcal{L}}^2$ for u	1.147 × 10${}^{-2}$	1.075 × 10${}^{-2}$	8.409 × 10${}^{-3}$	8.935 × 10${}^{-3}$	8.555 × 10${}^{-3}$	7.244 × 10${}^{-3}$
${\mathcal{L}}^2$ for f	1.562 × 10${}^{-1}$	1.552 × 10${}^{-1}$	1.488 × 10${}^{-1}$	1.462 × 10${}^{-1}$	1.474 × 10${}^{-1}$	1.524 × 10${}^{-1}$

shows the estimation values of Q and the relative error ${\mathcal{L}}^2$ for $u(t,x)$ and $f(t,x)$, where $(\alpha ,Q)$ is fixed to be (0.25, 1), (0.5, 1), (0.75, 1), (0.25, 2), (0.5,2) and (0.75,2), in turn. The experimental results show that the results of parameter estimation based on Gaussian processes have relatively high accuracy when fractional partial differential equations are considered. Furthermore, the posterior distribution of Gaussian processes can return a satisfactory numerical approximation to $u(t,x)$ and $f(t,x)$.

4.3. Simulation for a Partial Integro-Differential Equation

Example 3.

$$\begin{array}{c}\hfill {\mathcal{L}}_{(t,x)}^{\varphi }u(t,x):=\frac{\partial }{\partial t}u(t,x)-\alpha \Delta u(t,x)-\beta {\int }_0^t\Delta u(\tau ,x)d\tau =f,\end{array}$$

(22)

where $(t,x)\in [0,1]\times [0,1]$, $\varphi =(\alpha ,\beta )$, and $\Delta u(t,x)=\frac{{\partial }^{2}u}{\partial t^2}+\frac{{\partial }^{2}u}{\partial x^2}$. The exact solution is $u(t,x)=(1+t^3)(2-x^2)$, and the inhomogeneous term is $f(t,x)=\frac{1}{2}t[12t+\beta (4-12t+t^3)+6(-1+\beta )t{x}^2]+2\alpha [1+t^3+3t(-2+x^2)]$.

Experiments are performed with the noiseless data of Example 3.

Table 3. Partial integro-differential equation: estimated values of $(\alpha ,\beta )$ and the relative error ${\mathcal{L}}^2$ for $u(t,x)$ and $f(t,x)$, where $N_u=20$, $N_f=20$, $(\alpha ,\beta )$ is fixed to be (1, 1), (1, 2), (2, 1) and (2, 2), in turn.

$(\mathit{\alpha },\mathit{\beta })$	$(1,1)$	$(1,2)$	$(2,1)$	$(2,2)$
$\widehat{\alpha }$	1.000103	0.999282	2.000769	2.000163
$\widehat{\beta }$	0.999468	2.001618	0.996979	1.998910
${\mathcal{L}}^2$ for u	3.077 × 10${}^{-5}$	6.223 × 10${}^{-5}$	4.726 × 10${}^{-5}$	3.202 × 10${}^{-5}$
${\mathcal{L}}^2$ for f	6.331 × 10${}^{-3}$	4.784 × 10${}^{-3}$	5.294 × 10${}^{-3}$	3.857 × 10${}^{-3}$

shows the estimation values of $(\alpha ,\beta )$ and the relative error ${\mathcal{L}}^2$ for $u(t,x)$ and $f(t,x)$, where $(\alpha ,\beta )$ is fixed to be (1, 1), (1, 2), (2, 1) and (2, 2), in turn. The experimental results show that the results of parameter the estimation based on Gaussian processes have relatively high accuracy when partial integro-differential equations are considered. Furthermore, the posterior distribution of Gaussian processes can return a satisfying numerical approximation to $u(t,x)$ and $f(t,x)$.

Moreover, we investigate the impact of the amount of training data and the noise level on the accuracy of the estimation, where $(\alpha ,\beta )$ is taken as (1, 1).

Table 4. Numerical results for Example 3. Impact of the amount of training data: estimated values of $(\alpha ,\beta )$ and the relative error ${\mathcal{L}}^2$ for $u(t,x)$ and $f(t,x)$, where $(\alpha ,\beta )$ is fixed to be (1,1) and noise-free data is used.

${\mathit{N}}_{\mathit{u}}={\mathit{N}}_{\mathit{f}}$	10	20	30	40	50
$\widehat{\alpha }$	0.887239	1.000103	0.999763	1.000057	0.999051
$\widehat{\beta }$	1.343989	0.999468	1.001156	0.999610	1.005044
${\mathcal{L}}^2$ for u	1.979 × 10${}^{-2}$	3.077 × 10${}^{-5}$	6.761 × 10${}^{-6}$	2.510 × 10${}^{-6}$	2.752 × 10${}^{-5}$
${\mathcal{L}}^2$ for f	1.759 × 10${}^{-2}$	6.331 × 10${}^{-3}$	6.737 × 10${}^{-4}$	4.779 × 10${}^{-4}$	9.552 × 10${}^{-5}$

shows the estimation values of $(\alpha ,\beta )$ and the relative error ${\mathcal{L}}^2$ for $u(t,x)$ and $f(t,x)$, where $N_u$ and $N_f$ are fixed to be 10, 20, 30, 40 and 50, in turn. The results show that the larger amount of training data, the higher the prediction accuracy in general.

Table 5. Numerical results for Example 3. Impact of the levels of addictive noise: estimated values of $(\alpha ,\beta )$ and the relative error ${\mathcal{L}}^2$ for $u(t,x)$ and $f(t,x)$ where $N_u=N_f=20$, $(\alpha ,\beta )$ is fixed to be (1,1).

$({\mathit{\sigma }}_{\mathit{u}}^2,{\mathit{\sigma }}_{\mathit{f}}^2)$	$(0,0)$	$(0,0.5^2)$	$(0,1.0^2)$	$(0.{005}^2,0)$	$(0.{005}^2,0.5^2)$	$(0.{005}^2,1.0^2)$
$\widehat{\alpha }$	1.000103	0.983124	0.977075	0.974645	0.941651	0.926553
$\widehat{\beta }$	0.999468	1.125553	1.243469	1.450893	1.511768	1.582164
${\mathcal{L}}^2$ for u	3.077 × 10${}^{-5}$	1.467 × 10${}^{-3}$	1.684 × 10${}^{-3}$	4.594 × 10${}^{-3}$	4.274 × 10${}^{-3}$	5.241 × 10${}^{-3}$
${\mathcal{L}}^2$ for f	6.331 × 10${}^{-3}$	1.134 × 10${}^{-1}$	1.483 × 10${}^{-1}$	3.814 × 10${}^{-3}$	1.800 × 10${}^{-1}$	2.734 × 10${}^{-1}$

shows the estimation values and the relative error ${\mathcal{L}}^2$ where $N_u=N_f=20$, and the levels of addictive noise from the training data are taken at different values.

According to Table 5, we can conclude that the method can estimate parameters with relatively gratifying accuracy when noise-free data can not be obtained, although the estimation accuracy is sensitive to the noise of $u(t,x)$, which is likely due to the high complexity of the PDEs considered in this paper. Furthermore, we have to admit that this high sensitivity to the noise of the latent solution u greatly limits the scope of application of Gaussian processes. In summary, this problem deserves further research.

4.4. Simulation for a System of Partial Differential Equations

Example 4.

$$\begin{array}{c}\hfill \left\{\begin{array}{c}{\mathcal{L}}_{(t,x,y)}^{\varphi ,1}u+{\mathcal{L}}_{(t,x,y)}^{\varphi ,2}v=u_t+a{u}_{xx}+b{v}_{xy}=f_1,\hfill \\ {\mathcal{L}}_{(t,x,y)}^{\varphi ,3}u+{\mathcal{L}}_{(t,x,y)}^{\varphi ,4}v=u_{tt}+c{u}_{yy}+v_t+d{v}_{xy}=f_2,\hfill \end{array}\right.\end{array}$$

(23)

where $(t,x,y)\in {[0,1]}^3$, $\varphi =(a,b,c,d)$. $u(t,x,y)=e^{t}x(x-1)y(y-1)$, $v(t,x,y)=$$e^{t}sin\left(2\pi x\right)cos\left(2\pi y\right)$, $f_1(t,x,y)=e^t[(x^2-x+2a)(y^2-y)-4b{\pi }^{2}cos\left(2\pi x\right)sin\left(2\pi y\right)]$ and $f_2(t,x,y)=e^t[(x^2-x)(y^2-y+2c)+cos\left(2\pi y\right)sin\left(2\pi x\right)-4d{\pi }^{2}cos\left(2\pi x\right)sin\left(2\pi y\right)]$ satisfy Equation (23).

Experiments are performed with noiseless data of Example 4. Denote that the amount of training data of $u(t,x,y)$ is $N_u$, the amount of training data $v(t,x,y)$ is $N_v$, the amount of training data $f_1(t,x,y)$ is $N_{f_1}$, and the amount of training data $f_2(t,x,y)$ is $N_{f_2}$. Take $N_u=N_v=100$ and $N_{f_1}=N_{f_2}=80$ in the experiments.

Table 6. A system of partial differential equations: estimated values of (a, b, c, d) and the relative error ${\mathcal{L}}^2$ for u, v, $f_1$ and $f_2$, where $N_{u_1}=N_{u_2}=100$, $N_{f_1}=N_{f_2}=80$, and (a, b, c, d) is fixed to be (1, 1, 1, 1), (1, 1, 2, 2), (2, 2, 1, 1) and (2, 2, 2, 2), in turn.

(a, b, c, d)	$(1,1,1,1)$	$(1,1,2,2)$	$(2,2,1,1)$	$(2,2,2,2)$
$\widehat{a}$	1.000803	1.000828	2.001061	2.000685
$\widehat{b}$	1.000120	1.000112	2.000256	2.000255
$\widehat{c}$	0.999639	2.000954	0.998905	2.000404
$\widehat{d}$	0.999927	1.999879	0.999935	1.999898
${\mathcal{L}}^2$ for u	1.363 × 10${}^{-2}$	1.409 × 10${}^{-2}$	1.215 × 10${}^{-2}$	1.230 × 10${}^{-2}$
${\mathcal{L}}^2$ for v	4.805 × 10${}^{-3}$	4.601 × 10${}^{-3}$	4.754 × 10${}^{-3}$	4.395 × 10${}^{-3}$
${\mathcal{L}}^2$ for $f_1$	1.383 × 10${}^{-2}$	1.368 × 10${}^{-2}$	1.354 × 10${}^{-2}$	1.335 × 10${}^{-2}$
${\mathcal{L}}^2$ for $f_2$	1.396 × 10${}^{-2}$	1.374 × 10${}^{-2}$	1.366 × 10${}^{-2}$	1.339 × 10${}^{-2}$

shows the estimation values of (a, b, c, d) and the corresponding prediction errors for u, v, $f_1$ and $f_2$, where (a, b, c, d) is fixed to be (1, 1, 1, 1), (1, 1, 2, 2), (2, 2, 1, 1) and (2, 2, 2, 2), in turn. The experimental results show that the results of parameter estimation based on Gaussian processes have high accuracy, and the posterior distribution of Gaussian processes can return a satisfying numerical approximation to u, v, $f_1$ and $f_2$, when the system of linear PDEs is considered.

5. Conclusions

In this paper, we explored the possibility of using Gaussian processes in solving inverse problems of complex linear partial differential equations, which include high-order partial differential equations, partial integro-differential equations, fractional partial differential equations and a system of partial differential equations. The main points of the Gaussian processes method were to encode the distribution information of the data into kernels (covariance functions) of Gaussian process priors, transform unknown parameters of the linear operators into the hyper-parameters of the kernels, train the parameters and hyper-parameters through minimizing the negative log marginal likelihood and infer the solution of the considered equations.

Numerical experiments showed that the data-driven method based on Gaussian processes had high prediction accuracy when estimating the unknown parameters of the PDEs considered, which proved that Gaussian processes have impressive performance in dealing with linear problems. Furthermore, the posterior distribution of Gaussian processes can return a satisfactory numerical approximation to the latent solution and the inhomogeneous term of the PDEs. However, the estimation accuracy of unknown parameters was sensitive to the noise of the latent solution, which still deserves further research. In the future work, we may focus on how to exploit the Gaussian process to solve the inverse problem of nonlinear PDEs and on how to solve the problem of multi-parameter estimation for fractional partial differential equations.