Deep Theory of Functional Connections: A New Method for Estimating the Solutions of Partial Differential Equations

Abstract

This article presents a new methodology called Deep Theory of Functional Connections (TFC) that estimates the solutions of partial differential equations (PDEs) by combining neural networks with the TFC. The TFC is used to transform PDEs into unconstrained optimization problems by analytically embedding the PDE’s constraints into a “constrained expression” containing a free function. In this research, the free function is chosen to be a neural network, which is used to solve the now unconstrained optimization problem. This optimization problem consists of minimizing a loss function that is chosen to be the square of the residuals of the PDE. The neural network is trained in an unsupervised manner to minimize this loss function. This methodology has two major differences when compared with popular methods used to estimate the solutions of PDEs. First, this methodology does not need to discretize the domain into a grid, rather, this methodology can randomly sample points from the domain during the training phase. Second, after training, this methodology produces an accurate analytical approximation of the solution throughout the entire training domain. Because the methodology produces an analytical solution, it is straightforward to obtain the solution at any point within the domain and to perform further manipulation if needed, such as differentiation. In contrast, other popular methods require extra numerical techniques if the estimated solution is desired at points that do not lie on the discretized grid, or if further manipulation to the estimated solution must be performed.

1. Introduction

Partial differential equations (PDEs) are a powerful mathematical tool that is used to model physical phenomena, and their solutions are used to simulate, design, and verify the design of a variety of systems. PDEs are used in multiple fields including environmental science, engineering, finance, medical science, and physics, to name a few. Many methods exist to approximate the solutions of PDEs. The most famous of these methods is the finite element method (FEM) [1,2,3]. FEM has been incredibly successful in approximating the solution to PDEs in a variety of fields including structures, fluids, and acoustics. However, FEM does have some drawbacks.

FEM discretizes the domain into elements. This works well for low-dimensional cases, but the number of elements grows exponentially with the number of dimensions. Therefore, the discretization becomes prohibitive as the number of dimensions increases. Another issue is that FEM solves the PDE at discrete nodes, but if the solution is needed at locations other than these nodes, an interpolation scheme must be used. Moreover, extra numerical techniques are needed to perform further manipulation of the FEM solution.

Reference [4] explored the use of neural networks to solve PDEs, and showed that the use of neural networks avoids these problems. Rather than discretizing the entire domain into a number of elements that grows exponentially with the dimension, neural networks can sample points randomly from the domain. Moreover, once the neural network is trained, it represents an analytical approximation of the PDE. Consequently, no interpolation scheme is needed when estimating the solution at points that did not appear during training, and further analytical manipulation of the solution can be done with ease. Furthermore, Ref. [4] compared the neural network method with FEM on a set of test points that did not appear during training (i.e., points that were not the nodes in the FEM solution), and showed that the solution obtained by the neural network generalized well to points outside of the training data. In fact, the maximum error on the test set of data was never more than the maximum error on the training set of data. In contrast, the FEM had more error on the test set than on the training set. In one case, the test set had approximately three orders of magnitude more error than the training set. In short, Ref. [4] presents strong evidence that neural networks are useful for solving PDEs.

However, what was presented in Ref. [4] can still be improved. In Ref. [4] the boundary constraints are managed by adding extra terms to the loss function. An alternative method is to encapsulate the boundary conditions, by posing the solution in such a way that the boundary conditions must be satisfied, regardless of the values of the training parameters in the neural network. References [5,6] manage boundary constraints in this way when solving PDEs with neural networks by using a method similar to the Coons’ patch [7] to satisfy the boundary constraints exactly.

Exact boundary constraint satisfaction is of interest for a variety of problems, particularly when confidence in the constraint information is high. This is especially important for physics informed problems. Moreover, embedding the boundary conditions in this way means that the neural network needs to sample points from the interior of the domain only, not the domain and the boundary. While the methods presented in [5,6] work well for low-dimensional PDEs with simple boundary constraints, they lack a mechanized framework for generating expressions that embed higher-dimensional or more complex constraints while maintaining a neural network with free-to-choose parameters. For example, the fourth problem in the results section cannot be solved using the solution forms shown in Ref. [5,6]. Luckily, a framework that can embed higher-dimensional or more complex constraints has already been invented: The Theory of Functional Connections (TFC) [8,9]. In Ref. [10], TFC was used to embed constraints into support vector machines, but left embedding constraints into neural networks to future work. This research shows how to embed constraints into neural networks with the TFC, and leverages this technique to numerically estimate the solutions of PDEs. Although the focus of this article is a new technique for numerically estimating the solutions of PDEs, the article’s contribution to the machine learning community is farther reaching, as the ability to embed constraints into neural networks has the potential to improve performance when solving any problem that has constraints, not just differential equations, with neural networks.

TFC is a framework that is able to satisfy many types of boundary conditions while maintaining a function that can be freely chosen. This free function can be chosen, for example, to minimize the residual of a differential equation. TFC has already been used to solve ordinary differential equations with initial value constraints, boundary value constraints, relative constraints, integral constraints, and linear combinations of constraints [8,11,12,13]. Recently, the framework was extended to n-dimensions [9] for constraints on the value and arbitrary order derivative of $(n-1)$-dimensional manifolds. This means the TFC framework can now generate constrained expressions that satisfy the boundary constraints of multidimensional PDEs [14].

2. Theory of Functional Connections

The Theory of Functional Connections (TFC) is a mathematical framework designed to turn constrained problems into unconstrained problems. This is accomplished through the use of constrained expressions, which are functionals that represent the family of all possible functions that satisfy the problem’s constraints. This technique is especially useful when solving PDEs, as it reduces the space of solutions to just those that satisfy the problem’s constraints. TFC has two major steps: (1) Embed the boundary conditions of the problem into the constrained expression; (2) solve the now unconstrained optimization problem. The paragraphs that follow will explain these steps in more detail.

The TFC framework is easiest to understand when explained via an example, like a simple harmonic oscillator. Equation (1) gives an example of a simple harmonic oscillator problem.

$$m{\displaystyle \frac{{\mathrm{d}}^{2}y}{\mathrm{d}{x}_1^2}}+ky=0\mathrm{subject}\mathrm{to}:\left\{\begin{array}{c}y\left(0\right)=y_0\hfill \\ y_{x_1}\left(0\right)=y_{x_1},\hfill \end{array}\right.$$

(1)

where the subscript in $y_{x_1}$ denotes a derivative of y with respect to $x_1$.

Based on the univariate TFC framework presented in Ref. [8], the constrained expression is represented by the functional,

$$f(x_1,g\left(x_1\right))=g\left(x_1\right)+\sum _{j=1}^2{\eta }_j{s}_j\left(x_1\right),$$

where the $s_j\left(x_1\right)$ are a set of mutually linearly independent functions, called support functions, and ${\eta }_j$ are coefficient functions that are computed by imposing the constraints. For this example, let’s choose the support functions to be the first two monomials, $s_1\left(x_1\right)=1$ and $s_2\left(x_1\right)=x_1$. Hence, the constrained expression becomes,

$$f(x_1,g\left(x_1\right))=g\left(x_1\right)+{\eta }_1+x_1{\eta }_2.$$

(2)

The coefficient functions, ${\eta }_1\left(x_1\right)$ and ${\eta }_2\left(x_1\right)$, are solved by substituting the constraints into the constrained expression and solving the resultant set of equations. For the simple harmonic oscillator this yields the set of equations given by Equations (3) and (4).

$$\begin{array}{cc}\hfill y\left(0\right)& =g\left(0\right)+{\eta }_1\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill y_{x_1}\left(0\right)& =g_{x_1}\left(0\right)+{\eta }_2.\hfill \end{array}$$

(4)

Solving Equation (3) results in ${\eta }_1=y\left(0\right)-g\left(0\right)$, and solving Equation (4) yields ${\eta }_2=y_{x_1}\left(0\right)-g_{x_1}\left(0\right)$. Substituting ${\eta }_1$ and ${\eta }_2$ into Equation (3) we obtain,

$$f(x_1,g\left(x_1\right))=g\left(x_1\right)+y\left(0\right)-g\left(0\right)+x_1\left[y_{x_1}\left(0\right)-g_{x_1}\left(0\right)\right]$$

which is an expression satisfying the constraints, no matter what the free function, $g\left(x_1\right)$, is. In other words, this equation is able to reduce the solution space to just the space of functions satisfying the constraints, because for any function $g\left(x_1\right)$, the boundary conditions will always be satisfied exactly. Therefore, using constrained expressions transforms differential equations into unconstrained optimization problems.

This unconstrained optimization problem could be cast in the following way. Let the function to be minimized, $\mathcal{L}$, be equal to the square of the residual of the differential equation,

$$\mathcal{L}\left(x_1\right)={\left[m{\displaystyle \frac{{\mathrm{d}}^{2}f(x_1,g\left(x_1\right))}{\mathrm{d}{x}_1^2}}+kf\left(x_1\right)\right]}^2.$$

This function is to be minimized by varying the function $g\left(x_1\right)$. One way to do this is to choose $g\left(x_1\right)$ as a linear combination of a set of basis functions, and calculate the coefficients of the linear combination via least-squares or some other optimization technique. Examples of this methodology using Chebyshev orthogonal polynomials to obtain least-squares solutions of linear and nonlinear ordinary differential equations (ODEs) can be found in Refs. [11,12], respectively.

2.1. n-Dimensional Constrained Expressions

The previous example derived the constrained expression by creating and solving a series of simultaneous algebraic equations. This technique works well for constrained expressions in one dimension; however, it can become needlessly complicated when deriving these expression in n dimensions for constraints on the value and arbitrary order derivative of $n-1$ dimensional manifolds [9]. Fortunately, a different, more mechanized formalism exists that is useful for this case. The constrained expression presented earlier consists of two parts; the first part is a function that satisfies the boundary constraints, and the second part projects the free-function, $g\left(x_1\right)$, onto the hyper-surface of functions that are equal to zero at the boundaries. Rearranging Equation (2) highlights these two parts,

$$f(x_1,g\left(x_1\right))=\underset{A\left(x_1\right)}{\underbrace{y\left(0\right)+x_1{y}_{x_1}\left(0\right)}}+\underset{B(x_1,g\left(x_1\right))}{\underbrace{g\left(x_1\right)-g\left(0\right)-x_1{g}_{x_1}\left(0\right)}},$$

where $A\left(x_1\right)$ satisfies the boundary constraints and $B(x_1,g\left(x_1\right))$ is a functional projecting the free-function onto the hyper-surface of functions that are equal to zero at the boundaries.

The multivariate extension of this form for problems with boundary and derivative constraints in n-dimensions can be written compactly using Equation (5).

$$\begin{array}{cc}\hfill f(\mathbf{x},g(\mathbf{x}\left)\right)=& \underset{A\left(\mathbf{x}\right)}{\underbrace{{\mathcal{M}}_{i_1,i_2,\cdots ,i_n}\left(c\left(\mathbf{x}\right)\right)v_{i_1}\left(x_1\right)v_{i_2}\left(x_2\right)\cdots v_{i_n}\left(x_n\right)}}+\hfill \\ \hfill & +\underset{B(\mathbf{x},g(\mathbf{x}\left)\right)}{\underbrace{g\left(\mathbf{x}\right)-{\mathcal{M}}_{i_1,i_2,\cdots ,i_n}\left(g\left(\mathbf{x}\right)\right)v_{i_1}\left(x_1\right)v_{i_2}\left(x_2\right)\cdots v_{i_n}\left(x_n\right)}}\hfill \end{array}$$

(5)

where $\mathbf{x}={\{x_1,x_2,\cdots ,x_n\}}^T$ is a vector of the n independent variables, $\mathcal{M}$ is an n-th order tensor containing the boundary conditions $c\left(\mathbf{x}\right)$, the $v_{i_1},\cdots ,v_{i_n}$ are vectors whose elements are functions of the independent variables, $g\left(\mathbf{x}\right)$ is the free-function that can be chosen to be any function that is defined at the constraints, and $f(\mathbf{x},g(\mathbf{x}\left)\right)$ is the constrained expression. The first term, $A\left(\mathbf{x}\right)$, analytically satisfies the boundary conditions, and the term, $B(\mathbf{x},g(\mathbf{x}\left)\right)$, projects the free-function, $g\left(\mathbf{x}\right)$, onto the space of functions that vanish at the constraints. A mathematical proof that this form of the constrained expression satisfies the boundary constraints is given in Ref. [9]. The remainder of this section discusses how to construct the n-th order tensor $\mathcal{M}$ and the $\mathbf{v}$ vectors shown in Equation (5).

Before discussing how to build the $\mathcal{M}$ tensor and $\mathbf{v}$ vectors, let’s introduce some mathematical notation. Let $k\in [1,n]$ be an index used to denote the k-th dimension. Let ${}^k{c}_p^d:={\displaystyle \frac{{\partial }^{d}c\left(\mathbf{x}\right)}{\partial x_k^d}}{|}_{x_k=p}$ be the constraint specified by taking the d-th derivative of the constraint function, $c\left(\mathbf{x}\right)$, evaluated at the $x_k=p$ hyperplane. Further, let ${}^k{c}_{{\mathbf{p}}_k}^{{\mathbf{d}}_k}$ be the vector of ${\ell }_k$ constraints defined at the $x_k={\mathbf{p}}_k$ hyperplanes with derivative orders of ${\mathbf{d}}_k$, where ${\mathbf{p}}_k$ and ${\mathbf{d}}_k\in {\mathbb{R}}^{{\ell }_k}$. In addition, let’s define a boundary condition operator ${}^k{b}_p^d$ that takes the d-th derivative with respect to $x_k$ of a function, and then evaluates that function at the $x_k=p$ hyperplane. Mathematically,

$${}^k{b}_p^d\left[f\left(\mathbf{x}\right)\right]=\frac{{\partial }^{d}f\left(\mathbf{x}\right)}{\partial x_k^d}{|}_{x_k=p}.$$

This mathematical notation will be used to introduce a step-by-step method for building the $\mathcal{M}$ tensor. This step-by-step process will be be illustrated via a 3-dimensional example that has Dirichlet boundary conditions in $x_1$ and initial conditions in $x_2$ and $x_3$ on the domain $x_1,x_2,x_3\in [0,1]\times [0,1]\times [0,1]$. The $\mathcal{M}$ tensor is constructed using the following three rules.

• The element ${\mathcal{M}}_{111}=0$.
• The first order sub-tensor of $\mathcal{M}$ specified by keeping one dimension’s index free and setting all other dimension’s indices to 1 consists of the value 0 and the boundary conditions for that dimension. Mathematically,

  $${\mathcal{M}}_{1,\dots ,1,i_k,1,\dots ,1}=\left\{\begin{array}{c}0{,}^k{c}_{{\mathbf{p}}_k}^{{\mathbf{d}}_k}\end{array}\right\}.$$
  Using the example boundary conditions,

  $$\begin{array}{cc}\hfill & {\mathcal{M}}_{i_{1}11}={\left[0,c(0,x_2,x_3),c(1,x_2,x_3)\right]}^T\hfill \\ \hfill & {\mathcal{M}}_{1i_{2}1}={\left[0,c(x_1,0,x_3),c_{x_2}(x_1,0,x_3)\right]}^T\hfill \\ \hfill & {\mathcal{M}}_{11i_3}={\left[0,c(x_1,x_2,0),c_{x_3}(x_1,x_2,0)\right]}^T.\hfill \end{array}$$

  (6)
• The remaining elements of the $\mathcal{M}$ tensor are those with at least two indices different than one. These elements are the geometric intersection of the boundary condition elements of the first order tensors given in Equation (6), plus a sign (+ or −) that is determined by the number of elements being intersected. Mathematically this can be written as,

  $${\mathcal{M}}_{i_1{i}_2\dots i_n}={}^1{b}_{{\mathbf{p}}_{i_1-1}^1}^{{\mathbf{d}}_{i_1-1}^1}\left[{}^2{b}_{{\mathbf{p}}_{i_2-1}^2}^{{\mathbf{d}}_{i_2-1}^2}\left[\dots \left[{}^n{b}_{{\mathbf{p}}_{i_n-1}^n}^{{\mathbf{d}}_{i_n-1}^n}\left[c\left(\mathbf{x}\right)\right]\right]\dots \right]\right]{(-1)}^{m+1},$$

  where m is the number of indices different than one. Using the example boundary conditions we give three examples:

  $$\begin{array}{cc}\hfill & M_{133}=-c_{x_2{x}_3}(x_1,0,0)\hfill \\ \hfill & M_{221}=-c(0,0,x_3)\hfill \\ \hfill & M_{332}=c_{x_2}(1,0,0)\hfill \end{array}$$

A simple procedure also exists for constructing the $v_{i_k}$ vectors. The $v_{i_k}$ vectors have a standard form:

$$v_{i_k}={\left\{\begin{array}{ccccc}1,& {\displaystyle \sum _{i=1}^{{\ell }_k}{\alpha }_{i1}{h}_i\left(x_k\right),}& {\displaystyle \sum _{i=1}^{{\ell }_k}{\alpha }_{i2}{h}_i\left(x_k\right),}& \dots ,& {\displaystyle \sum _{i=1}^{{\ell }_k}{\alpha }_{i{\ell }_k}{h}_i\left(x_k\right)}\end{array}\right\}}^T,$$

where $h_i\left(x_k\right)$ are ${\ell }_k$ linearly independent functions. The simplest set of linearly independent functions, and those most often used in the TFC constrained expressions, are monomials, $h_i\left(x_k\right)=x_k^{i-1}$. The ${\ell }_k\times {\ell }_k$ coefficients, ${\alpha }_{ij}$, can be computed by matrix inversion,

$$\left[\begin{array}{cccc}{}^k{b}_{p_1}^{d_1}\left[h_1\right]& {}^k{b}_{p_1}^{d_1}\left[h_2\right]& \dots & {}^k{b}_{p_1}^{d_1}\left[h_{{\ell }_k}\right]\\ {}^k{b}_{p_2}^{d_2}\left[h_1\right]& {}^k{b}_{p_2}^{d_2}\left[h_2\right]& \dots & {}^k{b}_{p_2}^{d_2}\left[h_{{\ell }_k}\right]\\ ⋮& ⋮& \ddots & ⋮\\ {}^k{b}_{p_{{\ell }_k}}^{d_{{\ell }_k}}\left[h_1\right]& {}^k{b}_{p_{{\ell }_k}}^{d_{{\ell }_k}}\left[h_2\right]& \dots & {}^k{b}_{p_{{\ell }_k}}^{d_{{\ell }_k}}\left[h_{{\ell }_k}\right]\end{array}\right]\left[\begin{array}{cccc}{\alpha }_{11}& {\alpha }_{12}& \dots & {\alpha }_{1{\ell }_k}\\ {\alpha }_{21}& {\alpha }_{22}& \dots & {\alpha }_{2{\ell }_k}\\ ⋮& ⋮& \ddots & ⋮\\ {\alpha }_{{\ell }_{k}1}& {\alpha }_{{\ell }_{k}2}& \dots & {\alpha }_{{\ell }_k{\ell }_k}\end{array}\right]=\left[\begin{array}{cccc}1& 0& \dots & 0\\ 0& 1& \dots & 0\\ ⋮& ⋮& \ddots & ⋮\\ 0& 0& \dots & 1\end{array}\right].$$

Using the example boundary conditions, let’s derive the $v_{i_1}$ vector using the linearly independent functions $h_1=1$ and $h_2=x_1$.

$$\left[\begin{array}{cc}1& 0\\ 1& 1\end{array}\right]\left[\begin{array}{cc}{\alpha }_{11}& {\alpha }_{12}\\ {\alpha }_{21}& {\alpha }_{22}\end{array}\right]=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]\to \left[\begin{array}{cc}{\alpha }_{11}& {\alpha }_{12}\\ {\alpha }_{21}& {\alpha }_{22}\end{array}\right]=\left[\begin{array}{cc}1& 0\\ -1& 1\end{array}\right]$$

$$v_{i_1}={\left\{\begin{array}{ccc}1,& 1-x_1,& x_1\end{array}\right\}}^T.$$

For more examples and a mathematical proof that these procedures for generating the $\mathcal{M}$ tensor and the $\mathbf{v}$ vectors form a valid constrained expression see Ref. [9].

2.2. Two-Dimensional Example

This subsection will give an in depth example for a two-dimensional TFC case. The example is originally from problem 5 of Ref. [5], and is one of the PDE problems analyzed in this article. The problem is shown in Equation (7).

$$\begin{array}{cc}\hfill & {\nabla }^{2}z(x,y)=e^{-x}(x-2+y^3+6y)\mathrm{subject}\mathrm{to}\text{:}\hfill \\ \hfill & \left\{\begin{array}{c}z(x,0)=c(x,0)=x{e}^{-x}\hfill \\ z(0,y)=c(0,y)=y^3\hfill \\ z(x,1)=c(x,1)=e^{-x}(x+1)\hfill \\ z(1,y)=c(1,y)=(1+y^3)e^{-1}\hfill \end{array}\right.\hfill \\ \hfill & \mathrm{where}(x,y)\in [0,1]\times [0,1]\hfill \end{array}$$

(7)

Following the step-by-step procedure given in the previous section we will construct the $\mathcal{M}$ tensor:

• The first element is ${\mathcal{M}}_{11}=0$.
• The first order sub-tensors of $\mathcal{M}$ are:

  $$\begin{array}{cc}\hfill {\mathcal{M}}_{i_{1}1}& =\left\{\begin{array}{ccc}0& c(0,y)& c(1,y)\end{array}\right\}\hfill \\ \hfill {\mathcal{M}}_{1i_2}& =\left\{\begin{array}{ccc}0& c(x,0)& c(x,1)\end{array}\right\}\hfill \end{array}$$
• The remaining elements of $\mathcal{M}$ are the geometric intersection of elements from the first order sub-tensors.

  $$\begin{array}{cccc}\hfill {\mathcal{M}}_{22}& =-c(0,0)\hfill & \hfill & {\mathcal{M}}_{23}=-c(1,0)\hfill \\ \hfill {\mathcal{M}}_{32}& =-c(0,1)\hfill & \hfill & {\mathcal{M}}_{33}=-c(1,1)\hfill \end{array}$$

Hence, the $\mathcal{M}$ tensor is,

$${\mathcal{M}}_{i_1{i}_2}=\left[\begin{array}{ccc}0& c(0,y)& c(1,y)\\ c(x,0)& -c(0,0)& -c(1,0)\\ c(x,1)& -c(0,1)& -c(1,1)\end{array}\right]=\left[\begin{array}{ccc}0& y^3& (1+y^3)e^{-1}\\ x{e}^{-x}& 0& -e^{-1}\\ e^{-x}(x+1)& -1& -2e^{-1}\end{array}\right]$$

Following the step-by-step procedure given in the previous section we will construct the $\mathbf{v}$ vectors. For $v_{i_1}$, let’s choose the linearly independent functions $h_1=1$ and $h_2=x$.

$$\left[\begin{array}{cc}1& 0\\ 1& 1\end{array}\right]\left[\begin{array}{cc}{\alpha }_{11}& {\alpha }_{12}\\ {\alpha }_{21}& {\alpha }_{22}\end{array}\right]=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]\to \left[\begin{array}{cc}{\alpha }_{11}& {\alpha }_{12}\\ {\alpha }_{21}& {\alpha }_{22}\end{array}\right]=\left[\begin{array}{cc}1& 0\\ -1& 1\end{array}\right]$$

$$v_{i_1}={\left\{\begin{array}{ccc}1,& 1-x,& x\end{array}\right\}}^T.$$

For $v_{i_2}$ let’s choose the linearly indpendent functions $h_1=1$ and $h_2=y$.

$$\left[\begin{array}{cc}1& 0\\ 1& 1\end{array}\right]\left[\begin{array}{cc}{\alpha }_{11}& {\alpha }_{12}\\ {\alpha }_{21}& {\alpha }_{22}\end{array}\right]=\left[\begin{array}{cc}1& 0\\ 0& 1\end{array}\right]\to \left[\begin{array}{cc}{\alpha }_{11}& {\alpha }_{12}\\ {\alpha }_{21}& {\alpha }_{22}\end{array}\right]=\left[\begin{array}{cc}1& 0\\ -1& 1\end{array}\right]$$

$$v_{i_2}={\left\{\begin{array}{ccc}1,& 1-y,& y\end{array}\right\}}^T.$$

Now, we use the constrained expression form given in Equation (5) to finish building the constrained expression.

$$\begin{array}{cc}\hfill f(x,y,g(x,y\left)\right)=& g(x,y)+\frac{xy(y^2-1)}{e}+e^{-x}(x+y)+\hfill \\ \hfill & +(1-x)\left(g(0,0)+y\left(g(0,1)+y^2-g(0,0)-1\right)\right)+(x-1)g(0,y)+\hfill \\ \hfill & +x\left(yg(1,1)+(1-y)g(1,0)\right)-xg(1,y)+(y-1)g(x,0)-yg(x,1)\hfill \end{array}$$

(8)

Notice, that Equation (8) will always satisfy the boundary conditions of the problem regardless of the value of $g(x,y)$. Thus, the problem has been transformed into an unconstrained optimization problem where the cost function, $\mathcal{L}$, is the square of the residual of the PDE,

$$\mathcal{L}(x,y,g(x,y))={\left({\nabla }^{2}f(x,y,g(x,y))-e^{-x}(x-2+y^3+6y)\right)}^2.$$

For ODEs, the minimization of the cost function was accomplished by choosing g to be a linear combination of orthogonal polynomials with unknown coefficients, and performing least-squares or some other optimization technique to find the unknown coefficients. For two dimensions, one could make $g(x,y)$ the product of two linear combinations of these orthogonal polynomials, calculate all of the cross-terms, and then solve for the coefficients that multiply all terms and cross-terms using least-squares or non-linear least-squares. However, this will become computationally prohibitive as the dimension increases. Even at two dimensions, the number of basis functions needed, and thus the size of the matrix to invert in the least-squares, becomes large. An alternative solution, and the one explored in this article, is to make the free function, $g(x,y)$, a neural network.

3. PDE Solution Methodology

Similar to the last section, the easiest way to describe the methodology is with an example. The example used throughout this section will be the PDE given in Equation (7).

As mentioned previously, Deep TFC approximates solutions to PDEs by finding the constrained expression for the PDE and choosing a neural network as the free function. For all of the problems analyzed in this article, a simple, fully connected neural network was used. Each layer of a fully connected neural network consists of non-linear activation functions composed with affine transformations of the form $\mathcal{A}=W·\mathbf{x}+\mathbf{b}$, where W is a matrix of the neuron weights, $\mathbf{b}$ is a vector of the neuron biases, and $\mathbf{x}$ is a vector of inputs from the previous layer (or the inputs to the neural network if it is the first layer). Then, each layer is composed to form the entire network. For the fully connected neural networks used in this paper, the last layer is simply a linear output layer. For example, a neural network with three hidden layers that each use the non-linear activation function $\varphi$ can be written mathematically as,

$$\mathcal{N}(\mathbf{x};\theta )=W_4·\varphi \left(W_3·\varphi \left(W_2·\varphi \left(W_1·\mathbf{x}+{\mathbf{b}}_1\right)+{\mathbf{b}}_2\right)+{\mathbf{b}}_3\right)+{\mathbf{b}}_4,$$

where $\mathcal{N}$ is the symbol used for the neural network, $\mathbf{x}$ is the vector of inputs, $W_k$ are the weight matrices, ${\mathbf{b}}_k$ are the bias vectors, and $\theta$ is a symbol that represents all trainable parameters of the neural network; the weights and biases of each layer constitute the trainable parameters. Note that the notation $\mathcal{N}(x,y,\dots ;\theta )$ is also used in this paper for independent variables $x,y,\dots$ and trainable parameters $\theta$.

Thus, the constrained expression, given originally in Equation (8), now has the form given in Equation (9).

$$\begin{array}{cc}\hfill f(x,y;\theta )=& \mathcal{N}(x,y;\theta )+\frac{xy(y^2-1)}{e}+e^{-x}(x+y)+\hfill \\ \hfill & +(1-x)\left(\mathcal{N}(0,0;\theta )+y\left(\mathcal{N}(0,1;\theta )+y^2-\mathcal{N}(0,0;\theta )-1\right)\right)+(x-1)\mathcal{N}(0,y;\theta )+\hfill \\ \hfill & +x\left(y\mathcal{N}(1,1;\theta )+(1-y)\mathcal{N}(1,0;\theta )\right)-x\mathcal{N}(1,y;\theta )+(y-1)\mathcal{N}(x,0;\theta )-y\mathcal{N}(x,1;\theta )\hfill \end{array}$$

(9)

In order to estimate the solution to the PDE, the parameters of the neural network have to be optimized to minimize the loss function, which is taken to be the square of the residual of the PDE. For this example,

$${\displaystyle \mathcal{L}=\sum _i^N{\mathcal{L}}_i(x_i,y_i;\theta )\mathrm{where}{\mathcal{L}}_i(x_i,y_i;\theta )={\left({\nabla }^{2}f(x_i,y_i;\theta )-e^{-x_i}(x_i-2+y_i^3+6y_i)\right)}^2.}$$

The attentive reader will notice that training the neural network will require, for this example, taking two second order partial derivatives of $f(x,y;\theta )$ to calculate ${\mathcal{L}}_i$, and then taking gradients of $\mathcal{L}$ with respect to the neural network parameters, $\theta$, in order to train the neural network.

To take these higher order derivatives, TensorFlow’s™ gradients function was used [15]. This function uses automatic differentiation [16] to compute these derivatives. However, one must be conscientious when using the gradients function to ensure they get the desired gradients.

When taking the gradient of a vector, $y_j$, with respect to another vector, $x_i$, TensorFlow™ computes,

$${\displaystyle z_i={\displaystyle \frac{\partial }{\partial x_i}}\left(\sum _{j=1}^N{y}_j\right)}$$

where $z_i$ is a vector of the same size as $x_i$. The only example where it is not immediately obvious that this gradient function will give the desired gradient is when computing ${\nabla }^2{f}_i$. The desired output of this calculation is the following vector,

$$z_i={\left\{\begin{array}{ccc}{\displaystyle \frac{{\partial }^2{f}_1}{\partial x_1^2}}+{\displaystyle \frac{{\partial }^2{f}_1}{\partial y_1^2}},& \cdots ,& {\displaystyle \frac{{\partial }^2{f}_N}{\partial x_N^2}}+{\displaystyle \frac{{\partial }^2{f}_N}{\partial y_N^2}}\end{array}\right\}}^T,$$

where $z_i$ has the same size as $f_i$ and $(x_i,y_i)$ is the point used to generate $f_i$. TensorFlow’s™ gradients function will compute the following vector,

$${\tilde{z}}_i={\left\{\begin{array}{ccc}{\displaystyle \frac{{\partial }^2\left({\sum }_{j=1}^N{f}_j\right)}{\partial x_1^2}}+{\displaystyle \frac{{\partial }^2\left({\sum }_{j=1}^N{f}_j\right)}{\partial y_1^2}},& \cdots ,& {\displaystyle \frac{{\partial }^2\left({\sum }_{j=1}^N{f}_j\right)}{\partial x_N^2}}+{\displaystyle \frac{{\partial }^2\left({\sum }_{j=1}^N{f}_j\right)}{\partial y_N^2}}\end{array}\right\}}^T.$$

However, because $f_i$ only depends on the point $(x_i,y_i)$ and the derivative operator commutes with the sum operator, TensorFlow’s™ gradients function will compute the desired vector (i.e., ${\tilde{z}}_i=z_i$). Moreover, the size of the output vector will be correct, because the input vectors, $x_i$ and $y_i$, have the same size as $f_i$.

Training the Neural Network

Three methods were tested when optimizing the parameters of the neural networks:

• Adam optimizer [17]: A variant of stochastic gradient descent (SGD) that combines the advantages of two other popular SGD variants: AdaGrad [18] and RMSProp [19].
• Broyden–Fletcher–Goldfarb–Shanno [20] (BFGS): A quasi-Newton method designed for solving unconstrained, non-linear optimization problems. This method was chosen based on its performance when optimizing neural network parameters to estimate PDE solutions in Ref. [5].
• Hybrid method: Combines the first two methods by applying them in series.

For all four problems shown in this article, the solution error when using the BFGS optimizer was lower than with the other two methods. Thus, in the following section, the results shown use the BFGS optimizer.

The BFGS optimizer is a local optimizer, and the weights and biases of the neural networks are initialized randomly. Therefore, the solution error when numerically estimating PDEs will be different each time. However, Deep TFC guarantees that the boundary conditions are satisfied, and the loss function is the square of the residual of the PDE. Therefore, the loss function indicates how well Deep TFC is estimating the solution of the PDE at the training points. Moreover, because Deep TFC produces an analytical approximation of the solution, the loss function can be calculated at any point. Therefore, after training, one can calculate the loss function at a set of test points to determine whether the approximate solution generalizes well or has over fit the training points.

Due to the inherit stochasticity of the method, each Deep TFC solution presented in the results section that follows is the solution with the lowest mean absolute error of 10 trials. In other words, for each problem, the Deep TFC methodology was performed 10 times, and the best solution of those 10 trials is presented. Moreover, to show the variability in the Deep TFC method, problem 1 contains a histogram of the maximum solution error on a test set for 100 Monte Carlo trials.

4. Results

This section compares the estimated solution found using Deep TFC with with the analytical solution. Four PDE problems are analyzed. The first is the example PDE given in Equation (7), and the second is the wave equation. The third and fourth PDEs are simple solutions to the incompressible Navier–Stokes equations.

4.1. Problem 1

The first problem analyzed was the PDE given by Equation (7), copied below for the reader’s convenience.

$$\begin{array}{cc}\hfill & {\nabla }^{2}z(x,y)=e^{-x}(x-2+y^3+6y)\mathrm{subject}\mathrm{to}\text{:}\hfill \\ \hfill & \left\{\begin{array}{c}z(x,0)=x{e}^{-x}\hfill \\ z(0,y)=y^3\hfill \\ z(x,1)=e^{-x}(x+1)\hfill \\ z(1,y)=(1+y^3)e^{-1}\hfill \end{array}\right.\hfill \\ \hfill & \mathrm{where}(x,y)\in [0,1]\times [0,1]\hfill \end{array}$$

The known analytical solution for this problem is,

$$z=e^{-x}(x+y^3).$$

The neural network used to estimate the solution to this PDE was a fully connected neural network with 6 hidden layers, 15 neurons per layer, and 1 linear output layer. The non-linear activation function used in the hidden layers was the hyperbolic tangent. Other fully connected neural networks with various sizes and non-linear activation functions were tested, but this combination of size and activation function performed the best in terms of solution error. The biases of the neural network were all initialized as zero, and the weights were initialized using TensorFlow’s™ implementation of the Xavier initialization with uniform random initialization [21]. One hundred training points, $(x,y)$, evenly distributed throughout the domain were used to train the neural network.

Figure 1 shows the difference between the analytical solution and the estimated solution using Deep TFC on a grid of 10,000 evenly distributed points. This grid represents the test set. Figure 2 shows a histogram of the maximum solution error on the test set for 100 Monte Carlo trials.

The maximum error on the test set shown in Figure 1 was $2.780\times {10}^{-7}$ and the average error was $8.517\times {10}^{-8}$. Figure 2 shows that Deep TFC produces a solution at least as accurate as the solution in Figure 1 approximately 10% of the time. The remaining 90% of the time the solution error will be larger. Moreover, Figure 2 shows that the Deep TFC method is consistent. The maximum solution error in the 100 Monte Carlo tests was $3.891\times {10}^{-6}$, approximately an order of magnitude larger than the maximum solution error shown in Figure 1. The maximum error from Figure 1 is relatively low, six orders of magnitude lower than the solution values, which are on the order of ${10}^{-1}$.

Table 1. Comparison of Deep TFC, Ref. [5], and finite element method (FEM).

Method	Training Set	Test Set
Deep TFC	$3\times {10}^{-7}$	$3\times {10}^{-7}$
Ref. [5]	$5\times {10}^{-7}$	$5\times {10}^{-7}$
FEM	$2\times {10}^{-8}$	$1.5\times {10}^{-5}$

compares the maximum error on the test and training sets obtained with Deep TFC with the method used in Ref. [5] and FEM. Note, the FEM data was obtained from Table 1 of Ref. [5].

Table 1 shows that Deep TFC is slightly more accurate than the method from Ref. [5]. Moreover, in consonance with the findings from Ref. [5], the FEM solution performs better on the training set than Deep TFC, but worse on the solution set. Note also “that the accuracy of the finite element method decreases as the grid becomes coarser, and that the neural approach considers a mesh of $10\times 10$ points while in the finite element case a $18\times 18$ mesh was employed” [5].

The neural network used in this article is more complicated than the network used in Ref. [5], even though the two solution methods produce similarly accurate solutions. The constrained expression, $f(x,y;\theta )$, created using TFC, which is used as the assumed solution form, is more complex both in the number of terms and the number of times the neural network appears than the assumed solution form in Ref. [5]. For the reader’s reference, the assumed solution form for problem 1 from Ref. [5] is shown in Equation (10). Equation (10) was copied from Ref. [5], but the notation used has been transformed to match that of this paper; furthermore, a typo in the assumed solution form from Ref. [5] has been corrected here.

$$\begin{array}{cc}\hfill f(x,y;\theta )=& x(1-x)y(1-y)\mathcal{N}(x,y;\theta )+(1-x)y^3+x(1+y^3)e^{-1}\hfill \\ \hfill & +(1-y)x(e^{-x}-e^{-1})+y((1+x)e^{-x}-(1-x+2e^{-1}))\hfill \end{array}$$

(10)

To investigate how the assumed solution form affects the accuracy of the estimated solution, a comparison was made between the solution form from Ref. [5] and the solution form created using TFC in this article, while keeping all other variables constant. Furthermore, in this comparison, the neural network architecture used is identical to the neural network architecture given in Ref. [5] for this problem: one hidden layer with 10 neurons that uses a sigmoid non-linear activation function and a linear output layer. Each network was trained using the BFGS optimizer. The training points used were 100 evenly distributed points throughout the domain.

Figure 3 was created using the solution form posed in [5]. The maximum error on the test set was $4.246\times {10}^{-7}$ and the average error on the test set was $1.133\times {10}^{-7}$. Figure 4 was created using the Deep TFC solution form. The maximum error on the test set was $8.790\times {10}^{-6}$ and the average error on the test set was $2.797\times {10}^{-6}$.

Comparing Figure 3 and Figure 4 shows that the solution form from [5] gives an estimated solution that is approximately an order of magnitude lower in terms of average error and maximum error for this problem. Hence, the more complex TFC solution form requires a more complex neural network to achieve the same accuracy as the simpler solution form from Ref. [5] with a simple neural network. This results in a trade-off. The TFC constrained expressions allow for more complex boundary conditions (i.e., derivatives of arbitrary order) and can be used on n-dimensional domains, but require a more complex neural network. In contrast, the simpler solution form from Ref. [5] can achieve the same level of accuracy with a simpler neural network, but cannot be used for problems with higher order derivatives or n-dimensional domains.

4.2. Problem 2

The second problem analyzed was the wave equation, shown in Equation (11).

$$\begin{array}{cc}\hfill & \frac{{\partial }^{2}u}{\partial t^2}(x,t)=c^2\frac{{\partial }^{2}u}{\partial x^2}(x,t)\mathrm{subject}\mathrm{to}\text{:}\hfill \\ \hfill & \left\{\begin{array}{cc}\hfill & z(0,t)=0\hfill \\ \hfill & z(1,t)=0\hfill \\ \hfill & z(x,0)=x(1-x)\hfill \\ \hfill & z_t(x,0)=0\hfill \end{array}\right.\hfill \\ \hfill & \mathrm{where}(x,t)\in [0,1]\times [0,1]\hfill \end{array}$$

(11)

where the constant, $c=1$. The analytical solution for this problem is,

$$z(x,t)=\sum _{k=0}^{\infty }\frac{8}{{(2k+1)}^3{\pi }^3}sin\left((2k+1)\pi x\right)cos\left((2k+1)c\pi t\right).$$

Although the true analytical solution is an infinite series, for the purposes of making numerical comparisons, one can simply truncate this infinite series such that the error incurred by truncation falls below machine level precision. The constrained expression for this problem is shown in Equation (12).

$$\begin{array}{cc}\hfill f(x,t;\theta )=& (1-x)\left[\mathcal{N}(0,0;\theta )-\mathcal{N}(0,t;\theta )\right]+x\left[\mathcal{N}(1,0;\theta )-\mathcal{N}(1,t;\theta )\right]-\mathcal{N}(x,0;\theta )\hfill \\ \hfill & +x(1-x)+\mathcal{N}(x,t;\theta )+t\left[(1-x){\mathcal{N}}_t(0,0;\theta )+x{\mathcal{N}}_t(1,0;\theta )-{\mathcal{N}}_t(x,0;\theta )\right]\hfill \end{array}$$

(12)

The neural network used to estimate the solution to this PDE was a fully connected neural network with three hidden layers and 30 neurons per layer. The non-linear activation function used was the hyperbolic tangent. The biases and weights were initialized using the same method as problem 1. The training points, $(x,t)$, were created by choosing x to be an independent and identically distributed (IID) random variable with uniform distribution in the range $[0,1]$, and t to be an IID random variable with uniform distribution in the range $[0,1]$. The network was trained using the BFGS method and 1000 training points.

Figure 5 shows the difference between the analytical solution and the estimated solution using Deep TFC on a grid of 10,000 evenly distributed points; this grid represents the test set.

The maximum error on the test set was $2.643\times {10}^{-3}$ m and the average error on the test set was $6.305\times {10}^{-4}$ m. The error of this solution is larger than in the problem 1, while the solution values are on the same order of magnitude, ${10}^{-1}$ m, as in problem 1. The larger relative error in problem 2 is due to the more oscillatory nature of the solution (i.e., the surface of the true solution in problem 2 is more complex than that of problem 1).

4.3. Problem 3

The third problem analyzed was a known solution to the incompressible Navier–Stokes equations, called Poiseuille flow. The problem solves the flow velocity in a two-dimensional pipe in steady-state with a constant pressure gradient applied in the longitudinal axis. Equation (13) shows the associated equations and boundary conditions.

$$\begin{array}{cc}\hfill & \frac{\partial u}{\partial x}+\frac{\partial v}{\partial y}=0\hfill \\ \hfill & \rho \left(\frac{\partial u}{\partial t}+u\frac{\partial u}{\partial x}+v\frac{\partial u}{\partial y}\right)=-\frac{\partial P}{\partial x}+\mu \left(\frac{{\partial }^{2}u}{\partial x^2}+\frac{{\partial }^{2}u}{\partial y^2}\right)\hfill \\ \hfill & \rho \left(\frac{\partial v}{\partial t}+u\frac{\partial v}{\partial x}+v\frac{\partial v}{\partial y}\right)=\mu \left(\frac{{\partial }^{2}v}{\partial x^2}+\frac{{\partial }^{2}v}{\partial y^2}\right)\hfill \\ \hfill & \mathrm{subject}\mathrm{to}\text{:}\hfill \\ \hfill & \left\{\begin{array}{cc}\hfill & u(0,y,t)=u(L,y,t)=u(x,y,0)=\frac{1}{2\mu }\frac{\partial P}{\partial x}\left(y^2-{\left(\frac{H}{2}\right)}^2\right)\hfill \\ \hfill & u(x,\frac{H}{2},t)=u(x,-\frac{H}{2},t)=0\hfill \\ \hfill & v(0,y,t)=v(L,y,t)=v(x,y,0)=0\hfill \\ \hfill & v(0,\frac{H}{2},t)=v(0,-\frac{H}{2},t)=0\hfill \end{array}\right.\hfill \end{array}$$

(13)

where u and v are velocities in the x and y directions respectively, H is the height of the channel, P is the pressure, $\rho$ is the density, and $\mu$ is the viscosity. For this problem, the values $H=1$ m, $\rho =1$ kg/m³, $\mu =1$ Pa·s, and $\frac{\partial P}{\partial x}=-5$ N/m³ were chosen. The constrained expressions for the u-velocity, $f^u(x,y,t;\theta )$, and v-velocity, $f^v(x,y,t;\theta )$, are shown in Equation (14).

$$\begin{array}{cc}\hfill f^u(x,y,t;\theta )=& \mathcal{N}(x,y,t;\theta )-\mathcal{N}(x,y,0;\theta )+\frac{L-x}{L}\left(\mathcal{N}(0,y,0;\theta )-\mathcal{N}(0,y,t;\theta )\right)\hfill \\ \hfill & +\frac{x}{L}\left(\mathcal{N}(L,y,0;\theta )-\mathcal{N}(L,y,t;\theta )\right)+\frac{P\left(4y^2-H^2\right)}{8\mu }\hfill \\ \hfill & +\frac{1}{2HL}\left((2y-H)\right((L-x)\mathcal{N}\left(0,-\frac{H}{2},0;\theta \right)+x\mathcal{N}\left(L,-\frac{H}{2},0;\theta \right)-L\mathcal{N}\left(x,-\frac{H}{2},0;\theta \right)\hfill \\ \hfill & -(L-x)\mathcal{N}\left(0,-\frac{H}{2},t;\theta \right)+L\mathcal{N}\left(x,-\frac{H}{2},t;\theta \right)-x\mathcal{N}\left(L,-\frac{H}{2},t;\theta \right))\hfill \\ \hfill & -(H+2y)((L-x)\mathcal{N}\left(0,\frac{H}{2},0;\theta \right)-L\mathcal{N}\left(x,\frac{H}{2},0;\theta \right)+x\mathcal{N}\left(L,\frac{H}{2},0;\theta \right)\hfill \\ \hfill & -(L-x)\mathcal{N}\left(0,\frac{H}{2},t;\theta \right)-x\mathcal{N}\left(L,\frac{H}{2},t;\theta \right)+L\mathcal{N}\left(x,\frac{H}{2},t;\theta \right)\left)\right)\hfill \\ \hfill f^v(x,y,t;\theta )=& \mathcal{N}(x,y,t;\theta )-\mathcal{N}(x,y,0;\theta )+\frac{L-x}{L}\left(\mathcal{N}(0,y,0;\theta )-\mathcal{N}(0,y,t;\theta )\right)\hfill \\ \hfill & +\frac{x}{L}\left(\mathcal{N}(L,y,0;\theta )-\mathcal{N}(L,y,t;\theta )\right)+\frac{1}{2HL}\left((2y-H)\right((L-x)\mathcal{N}\left(0,-\frac{H}{2},0;\theta \right)\hfill \\ \hfill & +x\mathcal{N}\left(L,-\frac{H}{2},0;\theta \right)-L\mathcal{N}\left(x,-\frac{H}{2},0;\theta \right)-(L-x)\mathcal{N}\left(0,-\frac{H}{2},t;\theta \right)+L\mathcal{N}\left(x,-\frac{H}{2},t;\theta \right)\hfill \\ \hfill & -x\mathcal{N}\left(L,-\frac{H}{2},t;\theta \right))-(H+2y)((L-x)\mathcal{N}\left(0,\frac{H}{2},0;\theta \right)-L\mathcal{N}\left(x,\frac{H}{2},0;\theta \right)\hfill \\ \hfill & +x\mathcal{N}\left(L,\frac{H}{2},0;\theta \right)-(L-x)\mathcal{N}\left(0,\frac{H}{2},t;\theta \right)-x\mathcal{N}\left(L,\frac{H}{2},t;\theta \right)+L\mathcal{N}\left(x,\frac{H}{2},t;\theta \right)\left)\right)\hfill \end{array}$$

(14)

The neural network used to estimate the solution to this PDE was a fully connected neural network with four hidden layers and 30 neurons per layer. The non-linear activation function used was the sigmoid. The biases and weights were initialized using the same method as problem 1. The training points, $(x,y,t)$, were created by sampling x, y, and t IID from a uniform distribution that spanned the range of the associated independent variable. For x, the range was $[0,1]$. For y, the range was $[-\frac{H}{2},\frac{H}{2}]$, and for t, the range was $[0,1]$. The network was trained using the BFGS method on a batch size of 1000 training points. The loss function used was the sum of the squares of the residuals of the three PDEs in Equation (13).

The maximum error in the u-velocity was $3.308\times {10}^{-7}$ m per second, the average error in the u-velocity was $9.998\times {10}^{-8}$ m per second, the maximum error in the v-velocity was $5.575\times {10}^{-7}$ m per second, and the average error in the v-velocity was $1.542\times {10}^{-7}$ m per second. Despite the complexity, the maximum error and average error for this problem are six to seven orders of magnitude lower than the solution values. However, the constrained expression for this problem essentially encodes the solution, because the initial flow condition at time zero is the same as the flow condition throughout the spatial domain at any time. Thus, if the neural network outputs a value of zero for all inputs, the problem will be solved exactly. Although the neural network does output a very small value for all inputs, it is interesting to note that none of the layers have weights or biases that are at or near zero.

4.4. Problem 4

The fourth problem is another solution to the Navier–Stokes equations, and is very similar to the third. The only difference is that in this case, the fluid is not in steady state, it starts from rest. Equation (15) shows the associated equations and boundary conditions.

$$\begin{array}{cc}\hfill & \frac{\partial u}{\partial x}+\frac{\partial v}{\partial y}=0\hfill \\ \hfill & \rho \left(\frac{\partial u}{\partial t}+u\frac{\partial u}{\partial x}+v\frac{\partial u}{\partial y}\right)=-\frac{\partial P}{\partial x}+\mu \left(\frac{{\partial }^{2}u}{\partial x^2}+\frac{{\partial }^{2}u}{\partial y^2}\right)\hfill \\ \hfill & \rho \left(\frac{\partial v}{\partial t}+u\frac{\partial v}{\partial x}+v\frac{\partial v}{\partial y}\right)=\mu \left(\frac{{\partial }^{2}v}{\partial x^2}+\frac{{\partial }^{2}v}{\partial y^2}\right)\hfill \\ \hfill & \mathrm{subject}\mathrm{to}\text{:}\hfill \\ \hfill & \left\{\begin{array}{cc}\hfill & u(0,y,t)=\frac{\partial u}{\partial x}(L,y,t)=u(x,y,0)=0\hfill \\ \hfill & u(x,\frac{H}{2},t)=u(x,-\frac{H}{2},t)=0\hfill \\ \hfill & v(0,y,t)=\frac{\partial v}{\partial x}(L,y,t)=v(x,y,0)=0\hfill \\ \hfill & v(x,\frac{H}{2},t)=v(x,-\frac{H}{2},t)=0\hfill \end{array}\right.\hfill \end{array}$$

(15)

This problem was created to avoid encoding the solution to the problem into the constrained expression, as was the case in the previous problem. The constrained expressions for the u-velocity, $f^u(x,y,t;\theta )$, and v-velocity, $f^v(x,y,t;\theta )$, are shown in Equation (16).

$$\begin{array}{cc}\hfill f^u(x,y,t;\theta )=& \mathcal{N}(x,y,t;\theta )-\mathcal{N}(x,y,0;\theta )+\mathcal{N}(0,y,0;\theta )-\mathcal{N}(0,y,t;\theta )+x{\mathcal{N}}_x(L,y,0;\theta )-x{\mathcal{N}}_x(L,y,t;\theta )\hfill \\ \hfill & +\frac{1}{2H}\left((2y-H)\right(\mathcal{N}\left(0,-\frac{H}{2},0;\theta \right)-\mathcal{N}\left(x,-\frac{H}{2},0;\theta \right)+x{\mathcal{N}}_x\left(L,-\frac{H}{2},0;\theta \right)-\mathcal{N}\left(0,-\frac{H}{2},t;\theta \right)\hfill \\ \hfill & +\mathcal{N}\left(x,-\frac{H}{2},t;\theta \right)-x{\mathcal{N}}_x\left(L,-\frac{H}{2},t;\theta \right))-(H+2y)(\mathcal{N}\left(0,\frac{H}{2},0;\theta \right)-\mathcal{N}\left(x,\frac{H}{2},0;\theta \right)\hfill \\ \hfill & +x{\mathcal{N}}_x\left(L,\frac{H}{2},0;\theta \right)-\mathcal{N}\left(0,\frac{H}{2},t;\theta \right)+\mathcal{N}\left(x,\frac{H}{2},t;\theta \right)-x{\mathcal{N}}_x\left(L,\frac{H}{2},t;\theta \right)\left)\right)\hfill \\ \hfill f^v(x,y,t;\theta )=& \mathcal{N}(x,y,t;\theta )-\mathcal{N}(x,y,0;\theta )+\mathcal{N}(0,y,0;\theta )-\mathcal{N}(0,y,t;\theta )+x{\mathcal{N}}_x(L,y,0;\theta )-x{\mathcal{N}}_x(L,y,t;\theta )\hfill \\ \hfill & +\frac{1}{2H}\left((2y-H)\right(\mathcal{N}\left(0,-\frac{H}{2},0;\theta \right)-\mathcal{N}\left(x,-\frac{H}{2},0;\theta \right)+x{\mathcal{N}}_x\left(L,-\frac{H}{2},0;\theta \right)-\mathcal{N}\left(0,-\frac{H}{2},t;\theta \right)\hfill \\ \hfill & +\mathcal{N}\left(x,-\frac{H}{2},t;\theta \right)-x{\mathcal{N}}_x\left(L,-\frac{H}{2},t;\theta \right))-(H+2y)(\mathcal{N}\left(0,\frac{H}{2},0;\theta \right)-\mathcal{N}\left(x,\frac{H}{2},0;\theta \right)\hfill \\ \hfill & +x{\mathcal{N}}_x\left(L,\frac{H}{2},0;\theta \right)-\mathcal{N}\left(0,\frac{H}{2},t;\theta \right)+\mathcal{N}\left(x,\frac{H}{2},t;\theta \right)-x{\mathcal{N}}_x\left(L,\frac{H}{2},t;\theta \right)\left)\right)\hfill \end{array}$$

(16)

The neural network used to estimate the solution to this PDE was a fully connected neural network with four hidden layers and 30 neurons per layer. The non-linear activation function used was the hyperbolic tangent. The biases and weights were initialized using the same method as problem 1. Problem 4 used 2000 training points that were selected the same way as in problem 3, except the new ranges for the independent variables were $[0,15]$ for x, $[0,3]$ for t, and $[-\frac{H}{2},\frac{H}{2}]$ for y.

Figure 6, Figure 7 and Figure 8 show the u-velocity of the fluid throughout the domain at three different times. Qualitatively, the solution should look as follows. The solution should be symmetric about the line $y=0$, and the solution should develop spatially and temporally such that after a sufficient amount of time has passed and sufficiently far from the inlet, $x=0$, the u-velocity will be equal, or very nearly equal, to the steady state u-velocity of problem 3. Qualitatively, the u-velocity field looks correct in Figure 7 and Figure 8, and throughout most of the spatial domain in Figure 6. However, near the left end of Figure 6, the shape of the highest velocity contour does not match that of the other figures. This stems from the fact that none of the training points fell near this location. Other numerical estimations of this PDE were made with the exact same method, but with different sets of random training points, and in those that had training points near this location, the u-velocity matched the qualitative expectation. However, none of those estimated solutions had a quantitative u-velocity with an error as low as the one shown in Figure 6, Figure 7 and Figure 8. Quantitatively, the u-velocity at $x=15$ from Figure 8 was compared with the known steady state u-velocity, and had a maximum error of $5.378\times {10}^{-4}$ m per second and an average error of $3.117\times {10}^{-4}$ m per second.

5. Conclusions

This article demonstrated how to combine neural networks with the Theory of Functional Connections (TFC) into a new methodology, called Deep TFC, that was used to estimate the solutions of PDEs. Results on this methodology applied to four problems were presented that display how accurately relatively simple neural networks can approximate the solutions to some well known PDEs. The difficulty of the PDEs in these problems ranged from linear, two-dimensional PDEs to coupled, non-linear, three-dimensional PDEs. Moreover, while the focus of this article was on numerically estimating the solutions of PDEs, the capability to embed constraints into neural networks has the potential to positively impact performance when solving any problem that has constraints, not just differential equations, with a neural network.

Future work should investigate the performance of different neural network architectures on the estimated solution error. For example, Ref. [4] suggests a neural network architecture where the hidden layers contain element-wise multiplications and sums of sub-layers. The sub-layers are more standard neural network layers like the fully connected layers used in the neural networks of this article. Another architecture to investigate is that of extreme learning machines [22]. This architecture is a single layer neural network where the weights of the linear output layer are the only trainable parameters. Consequently, these architectures can ultimately be trained by linear or non-linear least squares for linear or non-linear PDEs respectively.

Another topic for investigation is reducing the estimated solution error by sampling the training points based on the loss function values for the training points of the previous iteration. For example, one could create batches where half of the new batch consists of half of the points in the previous batch that had the largest loss function value and the other half are randomly sampled from the domain. This should consistently give training points that are in portions of the domain where the estimated solution is farthest from the real solution.

Finally, future work will explore extending the hybrid systems approach presented in Ref. [23] to n-dimensions. Doing so would enable Deep TFC to solve problems that involve discontinuities at interfaces. For example, consider a heat conduction problem that involves two slabs of different thermal conductivities in contact with one another. At the interface condition, the temperature is continuous but the derivative of temperature is not.