Deep Neural Network Model for Approximating Eigenmodes Localized by a Confining Potential

Abstract

We study eigenmode localization for a class of elliptic reaction-diffusion operators. As the prototype model problem we use a family of Schrödinger Hamiltonians parametrized by random potentials and study the associated effective confining potential. This problem is posed in the finite domain and we compute localized bounded states at the lower end of the spectrum. We present several deep network architectures that predict the localization of bounded states from a sample of a potential. For tackling higher dimensional problems, we consider a class of physics-informed deep dense networks. In particular, we focus on the interpretability of the proposed approaches. Deep network is used as a general reduced order model that describes the nonlinear connection between the potential and the ground state. The performance of the surrogate reduced model is controlled by an error estimator and the model is updated if necessary. Finally, we present a host of experiments to measure the accuracy and performance of the proposed algorithm.

1. Introduction

In this paper we study features of the spectral problem for the family of elliptic operators of the reaction-diffusion type posed in the finite domain $\mathsf{\Omega }={\left[-B,B\right]}^n$, $B>0$. These operators are also known as Schrödinger operators or Schrödinger Hamiltonians and they are defined by the differential expression of the form:

$$H_{\omega }u=-\Delta u+V_{\omega }u.$$

Here $-\Delta$ is the distributional realisation of the Laplace operator and $V_{\omega }$ is the multiplication operator with the real function $V_{\omega }$. The parameter $\omega$ describes a random perturbation of a given potential. The associated spectral problem is to find an eigenvalue $\epsilon$ and an eigenmode $u\ne 0$ such that u verifies:

$$H_{\omega }u=\epsilon u$$

(1)

and a set of boundary conditions. We will consider boundary conditions that lead to the realization of the expression H as a self-adjoint operator in a Hilbert space. In particular, we will consider functions $u\in H_0^1\left(\mathsf{\Omega }\right)$ with Dirichlet boundary conditions and $u\in H_{\pi }^1\left(\mathsf{\Omega }\right)$, where $H_{\pi }^1\left(\mathsf{\Omega }\right)$ is the first-order Sobolev space of functions (square integrable functions whose gradient is also square integrable) that satisfy the periodic boundary conditions [1,2]. In what follows we will use $\parallel ·\parallel$ to denote the $L^2\left(\mathsf{\Omega }\right)$ norm of a square integrable function.

We will consider—as an academic prototype—short-range confining electrostatic potentials such as those considered in [3] (see also [1,2]) and a more challenging class of confining potentials related to the effect of Anderson localization [4]. By the effect of localization we mean that we search for eigenvalues $\epsilon$ such that u, $\parallel u\parallel =1$, is essentially zero in the large part of the domain $\mathsf{\Omega }$. The Anderson model describes quantum states of an electron in a disordered alloy that can transition from metallic to insulating behavior. Loosely stated, we aim to model the connection $\mathcal{E}:V_{\omega }\mapsto u_{\omega }$, where $u_{\omega }$, $u_{\omega }>0$ is the eigenmode of the lowermost $\epsilon$ in (1). Such eigenmodes are called the ground states, and ${\epsilon }_0=\epsilon$ is called the ground state energy. Note that for the elliptic reaction-diffusion operators $H_{\omega }$ defined in $L^2\left(\mathsf{\Omega }\right)$ and with the potential $V_{\omega }$, which is bounded, nonnegative and positive on a set of positive measures, there exists—by an application of the Krein–Rutman Theorem—the unique ground state $u_0>0$, which verifies (1) and so the mapping $\mathcal{E}$ is well defined; see [5,6].

We emphasize that we use the following regularization approach from [6] to deal with rough potentials. Namely for $V_{\omega }$, which is bounded, nonnegative and positive on a set of positive measures there exists $u_{\omega }\in C^1\left(\mathsf{\Omega }\right)$, $u_{\omega }>0$ such that $H_{\omega }{u}_{\omega }=\mathbf{1}$. It can be shown by direct calculation that the operator $A_{\omega }$ defined in $H_{\pi }^1\left(\mathsf{\Omega }\right)$ by the differential expression:

$$A_{\omega }\varphi =-\frac{1}{u_{\omega }^2}\mathrm{div}\left(u_{\omega }^2\mathrm{grad}\varphi \right)+\frac{1}{u_{\omega }}\varphi$$

has the same eigenvalues as the operator $H_{\omega }$. Furthermore, $\psi$ is an eigenmode of $H_{\omega }$ if and only if $\varphi =\psi /u_{\omega }$ is an eigenmode of $A_{\omega }$. Based on this equivalence, we call the function $W=1/u_{\omega }$ the effective confining potential defined by $V_{\omega }$. For more details see [5,7].

One possibility of obtaining data-sparse representations of the solutions of elliptic problems is through the use of tensor networks also known as tensor train decompositions or matrix product states [8,9]. We choose a more direct approach known as Variational Physical Informed Neural Network (VPINN) [10,11,12]. Realizations of these dense network architectures are trained to solve the variational formulation of the problem. This approach to training neural networks is a part of the unsupervised learning paradigm. It is a mesh-less approach that is capable of solving variational (physical) problems, by minimizing the loss functional, which combines the variational energy of the system together with penalty terms implementing further physical or normalization constraints.

The main physical constraint for the ground state is the positivity constraint. To deal with excited states we need to implement further symmetry constraints in the variational space. We opt for a different approach, also based on positivity constraints and a-posteriori error estimation. We use the neural network to approximate the solution of the source problem:

$$-\Delta u+Vu=\mathbf{1}$$

with associated boundary conditions. The solution u is called the landscape function, and in this case we are interested in the mapping $\mathcal{L}:V\mapsto u$. The landscape function is a positive function in $C^1\left(\mathsf{\Omega }\right)$ and its reciprocal $W=1/u$ is called the effective potential. The effective potential provides a mechanism that incurs localization on bounded states. To localize the excited states we use the approach of [5,7]. Let $W_{min,i}$ be the i-th lowermost minimum of the effective potential W. It was observed in [5] that the following heuristic formula:

$${\widetilde{ϵ}}_{i-1}=\left(1+{\displaystyle \frac{n}{4}}\right)W_{min,i}$$

(2)

yields good approximations to the energies of excited states. Note that this relationship, given its simplicity, is also something we might reasonably hope to learn algorithmically from a sample of landscape functions. This was stated as a motivation to utilize neural networks in the study of the eigenvalue problem for Schrödinger operators [3].

For an eigenmode $\psi$ of H with the eigenvalue $\epsilon$ we have the estimate: $\psi \left(x\right)\le {\epsilon u\parallel \psi \parallel }_{L^{\infty }\left(\mathsf{\Omega }\right)}$, $x\in \mathsf{\Omega }$. This estimate can be obtained (see [13]) using the Feynman–Kac formula for the representation of the bounded state as an expectation of an integral of the Brownian motion. It was further argued in [13] that an eigenmode $\psi$ with energy $\epsilon :=\epsilon \left(\psi \right)$ can only localize in the region:

$$\{x:\epsilon u(x)\ge 1\}.$$

(3)

Subsequently, as a combination of (2) and (3) we get both information on the excited state energy and information on the location of the excited state’s support.

The Motivation and the Contribution of this Paper

The use of neural networks as data-sparse representations of complex, high dimensional nonlinear mappings is an emerging trend in many disciplines. In particular, it has been used to tackle many body Schrödinger equations [14,15], the Black–Scholes equation, the Hamilton–Jacobi–Bellman equation, and the Allen–Cahn equation [10,11,16,17,18,19,20].

In general, all of the above problems can be reduced to computing an approximation of the function $u:\mathsf{\Omega }\subset {\mathbb{R}}^n\to \mathbb{R}$. This approximation is constructed by optimizing (training) the parameters of the family of test functions (we chose the family of all realizations of a given neural network architecture) so that the value of the appropriate energy functional (for the chosen model) is minimal. The main challenge in such an approach is to assess the approximation accuracy and to ensure that the computed realization of the neural network satisfies further physical constraints, such as symmetry constraint or the boundary conditions.

Further physical, but also numerical, constraints can be built into the optimization model in several ways. The most scalable and flexible way is to use penalization [10,11,20,21]. A alternative more subtle, and more accurate way is to introduce the constraints directly into the family of test functions as it is done in the architecture of the PauliNet from [14] (see also [22,23]), or to construct a family of test functions using an ansatz that combines several components of the solution, which are themselves realizations of neural networks [12,24].

In this study we focus on the potentials for which the Hamiltonian satisfies the Krein–Rutman theorem (the scaled ground state is the unique positive and smooth function). Examples of such potentials are the effective potentials associated with the Anderson localization. Since this is a more restrictive class of potentials than those considered in [14], we opt for a direct approach. Our contribution is the introduction of the residual error estimator into the Deep Ritz Algorithm from [20]. This in turn allows us to use Temple–Kato [25,26] or similar inequalities [27,28] to ascertain if the ground state generated by the neural network is a certified small perturbation of a physical eigenstate. For activation functions that are smooth enough we can calculate the strong form of the eigenvalue residual and then compute its norm using a quadrature or quasi-Monte Carlo integration. Using the residual estimator we stop the optimization (training process) when the eigenvalue residual is small enough (satisfies the preset tolerance) and/or the convergence criterion for the optimization algorithm is met (Adam optimizer). The use of ansatz functions based on neural networks, such as [24], will undoubtedly be a method of choice for 2D or 3D problems. However, this method depends on an accurate representation of the boundary of the domain $\mathsf{\Omega }$ and thus faces challenges in scaling to higher dimensions.

The treatment of physical symmetries becomes critical when approximating excited states. For dealing with this task, we reformulate the problem as an inverse problem based on the solution of the source problem $Hu=\mathbf{1}$. The main constraint that the solution must satisfy is again positivity, and we construct an error estimator to certify the quality of the solution.

The network architectures used so far are dense network architectures. Inspired by [3], we study a parameter-dependent family of potentials and present a fully convolutional encoder–decoder neural network as a reduced order (surrogate) model for this family of partial differential equations and the mapping $\mathcal{L}:V\mapsto u$. We formulate a new certified surrogate modeling approach based on neural networks that is inspired by the work on certified surrogate modeling from [29] and the U-net architecture from [30]. We use the residual error estimator from the first part of the paper as a criterion for the surrogate (encoder–decoder) model update. For further details see Section 3.5.

Let us summarize the three classes of exemplar problems studied in this paper. First, we study the eigenvalue problem for approximating the unique positive normalized ground state $u_0$. We aim to construct certified, robust and scalable—with respect to the increase in the dimension of the problem—approximation methods. Second, to approximate the eigenvalues higher in the spectrum we study the landscape function. The landscape function is obtained as a solution to the source problem $Hu=\mathbf{1}$. It is again a positive smooth function and positivity is the only physical constraint needed to study the localization phenomena for the associated eigenstates. Further, we use simple residual control to ensure that the computed solution is a small perturbation of the true landscape function. As the third class of problems, we present the data-based surrogate model of the map connecting a class of potentials to the associated landscape functions. Here we are concerned with the use of convolutional networks as a data-sparse reduced order model in the context of certified surrogate modeling of this mapping. In particular, we are interested in the possibility of updating the surrogate model based on the residual error estimator.

2. Theoretical Background

In order to be precise and explicit, we will present the theoretical foundations on a somewhat restricted set of neural network architectures. The network architectures that will be used in practical computations are presented in Appendix B. The change of the family of the realizations of neural networks over which the optimization is carried out does not change the presentation of the algorithms in any practical way. Our main contribution is in the introduction of the error control in the Deep Ritz algorithm from [20]. We will now summarize the basic definitions from [31], which are necessary to interpret the numerical experiments.

Definition 1.

Given $n_1,n_2,L\in \mathbb{N}$, a neural network θ of depth D with the input dimension $n_1$ and the output dimension $n_2$ is the D-tuple $\theta =\left((A_l,b_l):l=1,\cdots ,D\right)$ where:

$$(A_l,b_l)\in {\mathbb{R}}^{N_l\times N_{l-1}}\times {\mathbb{R}}^{N_l},l=1,\cdots ,D.$$

By the convention $n_1=N_0$ and $n_2=N_D$. In the case in which $D=2$ we call the network shallow, and otherwise the network is called deep. The vector $\overrightarrow{N}=\left(\begin{array}{ccc}{N}_0& \cdots & N_D\end{array}\right)$ is called the network architecture of the neural network θ.

We will use $D\left(\theta \right)$ to denote the depth of the given neural network $\theta$. In the case in which the structure of matrices $A_l$, $l=1,\cdots ,D$ is not further restricted, we say that the network is dense. In the case in which a sparsity pattern is assumed we have several subclasses of neural networks. For exemple, if the matrices $A_l$, $l=1,\cdots ,D$ have a structure of a Toeplitz matrix ${\left(A_l\right)}_{ij}={\left(w_{i-j}^l\right)}_{ij}$—here $w^l$, $l=1,\cdots ,D$ are parameter vectors defining a Toeplitz matrix [32]—we talk about convolutional neural networks.

Let $\rho :\mathbb{R}\to \mathbb{R}$ be a function that is not a polynomial. By $\rho$ we denote the function $\rho ={\otimes }_{l=1}^n\rho :{\mathbb{R}}^n\to {\mathbb{R}}^n$. We will now define a realization of the neural network $\theta$ with respect to the function $\rho$.

Definition 2.

A function $R_{\theta ,\rho }:{\mathbb{R}}^n\to {\mathbb{R}}^m$ is defined by the formula:

$$R_{\theta ,\rho }\left(x\right)=T_L\left(\rho \left(T_{D-1}\left(\rho \left(T_{D-2}(\cdots \rho \left(T_1\left(x\right)\right))\right)\right)\right)\right).$$

where $T_l\left(x\right)=A_{l}x+b_l$, $l=1,\cdots ,D$ is called the realization of the neural network θ with respect to the activation function ρ.

Among various activation functions we single out the rectified linear unit (ReLU) function ${\rho }_{LU}=max\{0,x\}$ and the sigmoid function ${\rho }_S\left(x\right)=1/(1-exp(-x))$. The set of all ReLU realizations of a neural network $\theta$ has a special structure. We call a function $f:{\mathbb{R}}^{n_1}\to {\mathbb{R}}^{n_2}$ piece-wise linear if there is a finite set of pairwise disjoint, closed polyhedra whose union is ${\mathbb{R}}^{n_1}$ such that a restriction of f onto a chosen polyhedron is an affine function. It has been shown in [33] that any piece-wise linear function can be represented by a ReLU neural network and that any ReLU realization of a neural network is piece-wise linear. This observation is key to linking the approximation theory for neural networks with the standard Sobolev space regularity theory for partial differential equations.

Let us now fix some further notation. Let m be the number of the degrees of freedom of the space of piece-wise linear functions associated to the fixed polyhedral tessellation of $\mathsf{\Omega }$. We use $V_m$ to denote the set of all piece-wise linear functions on this tessellation. We also use the notations $P_1$, $P_2$ and $P_3$ to denote the space of piece-wise linear, quadratic and cubic functions, respectively. The corresponding interpolation operators (for continuous arguments) are denoted respectively by $I_{P_1}$, $I_{P_2}$ and $I_{P_3}$.

We will now briefly review a-posteriori error estimates that are used in this work. Let us note the following convention. We use ${\epsilon }_0$ and $u_0$ to denote the ground state energy and the normalized positive ground state. We use ${\epsilon }_1$ to denote the energy of a first excited state and we note that the notation is generalized for higher excited states in an obvious way. We denote the Rayleigh quotient of the operator H for the state $\psi$ by $\epsilon =\epsilon \left(\psi \right):=(\psi ,H\psi )/(\psi ,\psi )$. The standard Kato–Temple estimate from [26] can be written in a dimension-free form, also known as the relative form:

$$\frac{|\epsilon -{\epsilon }_0|}{\left|\epsilon \right|}\le \frac{\left|\epsilon \right|}{|{\epsilon }_1-\epsilon |}\frac{{\parallel H\psi -\epsilon \psi \parallel }^2}{{\left|\epsilon \right|}^2{\parallel \psi \parallel }^2}.$$

(4)

The quantity $\left|\epsilon \right|/|{\epsilon }_1-\epsilon |$ is a measure of the so-called relative spectral gap [34,35] and it measures the distance to the unwanted part of the spectrum. It can be estimated by symmetry considerations or other a-priori information. In fact, a more careful analysis from [35] shows that the scaled residual is an asymptotically exact estimate of the relative error and so we will heuristically drop the measure of the gap even in the preasymptotic regime. A consequence of the Davis–Kahan theorem [36] is that the residual also estimates the eigenvector error:

$$\frac{\parallel \psi -u_0\parallel }{\parallel \psi \parallel }\le c\frac{\left|\epsilon \right|}{|{\epsilon }_1-\epsilon |}\frac{\parallel H\psi -\epsilon \psi \parallel }{\left|\epsilon \right|\parallel \psi \parallel }.$$

(5)

Subsequently, the error estimator ${\Delta }_{\epsilon }^2={\parallel H\psi -\epsilon \psi \parallel }^2/({\epsilon }^2{\parallel \psi \parallel }^2)$ is a good stopping criterion for a certified approximation of an eigenmode.

For the source problem $Hu=\mathbf{1}$, $u\in H_0^1\left(\mathsf{\Omega }\right)$ we note the following relationship:

$$\frac{\parallel u-\underline{u}{\parallel }_{L^2}}{\parallel \underline{u}{\parallel }_{L^2}}\le \frac{{\parallel \mathbf{1}\parallel }_{L^2}}{{\epsilon }_0}\frac{\parallel H\underline{u}{-\mathbf{1}\parallel }_{L^2}}{\parallel \underline{u}{\parallel }_{L^2}{\parallel \mathbf{1}\parallel }_{L^2}},$$

between the residual and the relative $L^2$ error. Subsequently we use $\tau =\parallel H\underline{u}{-\mathbf{1}\parallel }_{L^2}$/ $(\parallel \underline{u}{\parallel }_{L^2}\parallel \mathbf{1}{\parallel }_{L^2})$ as an error estimator for the source problem.

Algorithms

We will now present the modifications of algorithms that we used to study the localization phenomena. We modified the Deep Ritz algorithm from [20] with the introduction of the a-posteriori (residual) error estimator. We call our variant the Certified Deep Ritz Algorithm. It is motivated by the work on certified reduced-order modeling in [29]. In order to be able to formulate strong residuals, we chose smooth activation functions $\rho$, so that $R_{\theta ,\rho }$ can be used to form the strong residual.

The performance of the stochastic gradient descent, when applied to the loss function, can be highly sensitive to the choice of the learning rate. Furthermore, it can also suffer from oscillation effects introduced by the choice of the sampling method in the numerical integration routines. For this reason we have opted to use the Adam optimizer from [37], which determines the learning rate by adaptively using information from higher-order moments.

To enforce the positivity constraint, we composed a realization of the neural network with the function $\Xi$, $\Xi >0$. We call the function $\Xi$ a positivity mask and it must be chosen appropriately for the governing boundary conditions. As the positivity mask for Schrödinger Hamiltonians we either chose a smooth nonnegative function $\Xi$, which decays to zero as $\left|x\right|\to \infty$, or set the positivity mask as the identity.

In Algorithm 1 the parameter $\beta >0$ is the penalty parameter used to enforce the boundary conditions and the parameter $\eta >0$ is used to normalize the eigenmode approximation. We solve the integral using a Gauss quadrature rule in 1D and for higher dimensional problems we use quasi-Monte Carlo integration from [38,39] or a sparse grid quadrature [40] to approximate the integrals in the loss function as well as for the final (more accurate) evaluation of the energy functional. Alternatively in 2D, we sometimes choose to compute the integrals by projecting the realization of a neural network into a finite element space and then use the finite element quadrature to compute the integral. According to the authors of [11,41], this is an appropriate approach for problems of moderate dimensions ($\mathsf{\Omega }\subset {\mathbb{R}}^n$, $n\le 20$). For higher-dimensional problems, Monte Carlo integration is the only scalable approach recommended.

Algorithm 1: Certified Deep Ritz Algorithm.

To apply Algorithm 1 to an eigenvalue problem we choose:

$$\tilde{\Delta }:={\Delta }_{\epsilon }^2=\parallel H{\psi }_{\theta ,\rho }-\epsilon \left({\psi }_{\theta ,\rho }\right){\psi }_{\theta ,\rho }{\parallel }^2/(\epsilon {\left({\psi }_{\theta ,\rho }\right)}^2\parallel {\psi }_{\theta ,\rho }{\parallel }^2)$$

and set the normalization parameter $\eta >0$ and $\beta >0$ for the loss function:

$$L(u;\beta ,\eta )={\displaystyle \frac{{\int }_{\mathsf{\Omega }}{|\nabla u|}^2+{\int }_{\mathsf{\Omega }}{V}_{\omega }{u}^2}{{\int }_{\mathsf{\Omega }}u}}+\beta {\parallel u\parallel }_{L^2(\partial \mathsf{\Omega })}^2+\eta \left({\int }_{\mathsf{\Omega }}{u}^2-1\right).$$

(6)

In the case of the source problem for the computation of the landscape function we set the normalization parameter $\eta =0$ and $\beta >0$ and define the loss function as:

$$L(u;\beta ,\eta )=\frac{1}{2}\left({\int }_{\mathsf{\Omega }}{|\nabla u|}^2+{\int }_{\mathsf{\Omega }}{V}_{\omega }{u}^2\right)-{\int }_{\mathsf{\Omega }}u+\beta {\parallel u\parallel }_{L^2(\partial \mathsf{\Omega })}^2.$$

As the error indicator we take $\tilde{\Delta }=\tau =\parallel H\underline{u}{-\mathbf{1}\parallel }_{L^2}/(\parallel \underline{u}{\parallel }_{L^2}{\parallel \mathbf{1}\parallel }_{L^2})$, where $\underline{u}=\Xi \circ R_{\theta ,\rho }$. Here we have chosen as an example the homogeneous Dirichlet boundary condition $u{|}_{\partial \mathsf{\Omega }}=0$. Other self-adjoint boundary conditions can equally be implemented by penalizing the boundary conditions residual at the boundary $\partial \mathsf{\Omega }$. Note that computing derivatives of realizations of neural networks is efficiently implemented in many programming environments such as TensorFlow [42].

3. Results

In this section we present direct approximation methods for estimating the ground state $u_0$, the ground state energy ${\epsilon }_0$ and the landscape function u. We use the Certified Deep Ritz Algorithm presented as Algorithm 1.

3.1. Direct Approximations of the Ground State in 1D

We now present the results of the application of Algorithm 1 on the 1D Schrödinger operator $H=-{\partial }_{xx}+V.$ For the domain $\mathsf{\Omega }=(-B,B)$, we choose the loss function (6) and set $\Xi \left(x\right)=exp(-x^2/10)$ as the positivity mask in Algorithm 1.

To benchmark the accuracy of the VPINN approximations we have solved the problem to high relative accuracy using the Chebyshev spectral method as implemented in the package chebfun [43,44]. We emphasize that chebfun was not used during the training of the network in any way. To compute the residuals and the energy of the ground state we used a Gaussian quadrature where the deep network is evaluated at the sufficient number—for the given interval $(-B,B)$—of Gaussian points.

We constructed the potential V as a linear combination of the finite well and two inverted Gaussian bell functions:

$$V=-{\alpha }_{1}exp(-\parallel ·-c_1{\parallel )}^2/k_1^2)-{\alpha }_{2}exp(-\parallel ·-c_2{\parallel )}^2/k_2^2)-h{\mathbf{1}}_{\{x:|x-c|<t\}}.$$

We used ${\mathbf{1}}_{\{x:|x-c|<t\}}$ to denote the indicator function and chose ${\alpha }_i\in [8,12]$, $c_i\in [-2.5,2.5]$, $k_i\in [0.9,2.6]$, $i=1,2$, $h\in [10,15]$, $c\in [-2,2]$, $t\in [0.5,2.5]$. The neural network has 1162 trainable parameters and we used the DenseNet VPINN architecture ${\overrightarrow{N}}_{\mathtt{DenseNet}}(1,4,2,10)$ with the activation function $\rho \left(x\right)=exp(-x^2/10)$; see Figure A1. Figure 1 shows the solution and the error estimate during 15,000 epochs of a run of the Adam optimizer with the learning rate $\delta ={10}^{-3}$ and a batch size of 1024. In this example we used 1024 quadrature points on an interval $(-6,6)$ and the penalty parameters $\beta ={10}^{-3}$ and $\eta =20$.

One can observe robust, almost asymptotically exact, performance of the estimator:

$${\Delta }_{\epsilon }^2={\parallel H\psi -\epsilon \psi \parallel }^2/({\epsilon }^2{\parallel \psi \parallel }^2).$$

Let us also emphasize that ${\Delta }_{\epsilon }$ measures the distance of the Rayleigh quotient (energy functional) to the nearest eigenvalue. For evaluation of the integrals in higher dimensions we refer the reader to Appendix B. Note that the final error for the approximation of the ground state energy was $0.1\%$, whereas the final relative $L^2$ error in the ground state was $0.9\%$. This is in line with the eigenmode error estimate (5).

3.2. Direct Approximations of the Ground State in Higher-Dimensional Spaces

We present VPINN approximation results for the ground state and the ground state energy of the Schrödinger equation with harmonic oscillator potential $V\left(x\right)={\parallel x\parallel }^2$ using the dense network with the architecture depicted in Figure A1. We study the problem on the truncated domain ${[-3,3]}^n$, where n is the dimension of the space. Neural network architecture should be constructed with caution. There are multiple sources of instability when dealing with neural networks, e.g., exploding and vanishing gradients. We experimented with a variety of different activation functions: ${\rho }_S\left(x\right)$, ${\rho }_{LU}{(x/10)}^2$, $x{\rho }_S\left(x\right)$, $exp(-x^2/10)$ etc. After training of the neural network using the quasi-Monte Carlo realization of the energy integrals to define the loss function we computed the approximate ground state energy using the approximation of the energy functional (Rayleigh quotient) using the Sobol sequences with 100,000 points. We also report on the results obtained using Smolyak grids of order 6 with the Gauss–Patterson rule. The results are presented in

Table 1. Convergence rates for the ground state energy of the harmonic oscillator in relation to the dimension. QMC: quasi-Monte Carlo.

n	${\mathit{\epsilon }}_0$	M for the Loss Function	Adam Optimizer Epochs	M for the Smolyak Quadrature	Smolyak Relative Error %	Relative Error for QMC with $\mathit{M}={10}^5$ Points%
1	1	100	50,000	127	0.004	0.003
2	2	1000	20,000	769	1.416	1.226
3	3	5000	50,000	2815	1.110	1.608
6	6	50,000	80,000	40,193	-	1.40
9	9	50,000	50,000	242,815	230.366	5.816

.

The accuracy of the ground state energy approximations that were obtained using quasi-Monte Carlo integration are comparable with the accuracy reported in [20]. On the other hand, the results obtained by using Smolyak’s points were unsatisfactory in dimensions higher than 3. This observation will be the subject of future research. It appears that the oscillation of the realizations of the neural network on the boundary of the computational domain together with the appearance of negative weights in sparse grid integral formulas contributed to the instability of the approach. We used the $\overrightarrow{N}=(n,5,2,20)$ architectures for $n\le 6$ and $\overrightarrow{N}=(9,3,1,10)$ in 9D and the swish function $\rho \left(x\right)=x{\rho }_S\left(x\right)$ as the activation function. The positivity mask was chosen as the identity.

3.3. Approximations of the Landscape Function in 1D

We now present the result of the approximation method using the landscape function as a solution of the partial differential equation $Hu=\mathbf{1}$, $u\in H_0^1\left(\mathsf{\Omega }\right)$. The potential $V\in C\left(\left[-50,50\right]\right)$ is constructed as a random piece-wise linear function (see Figure 2). More to the point, let ${\xi }_k$, $k=-50,\cdots ,50$ be independently drawn numbers from the uniform distribution on the interval $\left[0,4\right]$. We construct the potential V as the piece-wise linear interpolant of $(k,{\xi }_k)$, $k=-50,\cdots ,50$. We again used chebfun for benchmarking. We set $\eta =0$ and defined the loss function as:

$$L(u;\beta ,\eta )={\displaystyle \frac{1}{2}}\left({\int }_{\mathsf{\Omega }}{|\nabla u|}^2+{\int }_{\mathsf{\Omega }}{V}_{\omega }{u}^2\right)-{\int }_{\mathsf{\Omega }}u+\beta (u{(-50)}^2+u{\left(50\right)}^2).$$

(7)

Here we have used the architecture of the dense VPINN network; see Appendix C. We used $\beta =500$ for the experiments. In Figure 2 we can see the six local minima of the effective potential $W=1/u$ obtained from the neural network and the first six eigenstates computed by the chebfun. Note that the potential $W=1/u$ is defined only in the interior of the domain $(-50,50)$. In

Table 2. We tested the accuracy of the predictor ${\widetilde{ϵ}}_{i-1}=\left(1+{\displaystyle \frac{n}{4}}\right)W_{min,i}$ for 16 lowermost eigenvalues. The chebfun solution was used to benchmark the error.

	0	1	2	3	4	5	6	7
Minimum values of W	0.747201	0.918677	0.918754	0.933014	1.028903	1.057663	1.174706	1.245278
chebfun eigenvalues	0.979730	1.071839	1.230230	1.282611	1.301724	1.485232	1.577349	1.588252
Relative error in %	4.6675	7.1379	6.6481	9.0708	1.1981	1.9850	6.9082	1.9930
	8	9	10	11	12	13	14	15
Minimum values of W	1.256498	1.273980	1.326926	1.613203	1.848415	1.868003	1.907063	1.931723
chebfun eigenvalues	1.625253	1.758768	1.780166	2.095899	2.161778	2.265704	2.270798	2.278380
Relative error in %	3.3614	9.4551	6.8257	3.7882	6.8805	3.05864	4.9776	5.9811

we present the results of the benchmarking of the approximation formula (2) against highly accurate chebfun eigenvalue approximations.

The neural network has 6402 trainable parameters and we used the VPINN architecture ${\overrightarrow{N}}_{\mathtt{DenseNet}}=(1,5,2,20)$ with the activation function ${\rho }_{LU}{(x/10)}^2$. The positvitiy mask was chosen as the identity. The network was trained using 50,000 epochs of the Adam optimizer with the learning rate $\delta ={10}^{-3}$ and a batch size of 2048. In this example we also used 2048 quadrature points on an interval $(-50,50)$.

3.4. Direct VPINN Approximation of the Landscape Function in 2D

We now apply Algorithm 1 to the problem of approximating the landscape function in 2D. When presenting the examples we will report on the used network architecture ${\overrightarrow{N}}_{\mathtt{DenseNet}}=(2,k,l,m)$ as well as indicate the number of trainable parameters for each of the architectures. The activation function used for all neural networks in this subsection is the sigmoid function ${\rho }_S$. We now set $l=2$ and in the next table present a convergence study for the family of architectures ${\overrightarrow{N}}_{\mathtt{DenseNet}}=(2,k,2,m)$.

The convergence histories of relative $L^2$ and $H^1$ errors, measured with respect to the benchmark FEniCS solution, are shown in

Table 3. A report on the convergence in k and m for the family of architectures ${\overrightarrow{N}}_{\mathtt{DenseNet}}=(2,k,2,m)$. We benchmark the error against the highly accurate $P_3$ FEniCS solution.

Parameters	k	m	Relative ${\mathit{L}}^2$ Error 100,000 Epoch	Relative ${\mathit{H}}^1$ Error 100,000 Epoch	Relative ${\mathit{L}}^2$ Error 200,000 Epoch	Relative ${\mathit{H}}^1$ Error 200,000 Epoch	Relative Error of the First Three Eigenvalues Respectively
803	4	8	2.5852%	5.6216%	2.0527%	4.9876%	0.1638%, 1.4479%, 1.1472%
1203	4	10	2.7487%	5.3611%	1.2354%	3.6960%	0.0839%, 2.3489%, 0.6341%
1753	5	10	1.9314%	4.2386%	1.0679%	3.3851%	0.5957%, 1.9264%, 0.3822%
2403	6	10	1.1745%	3.0548%	0.7998%	2.6994%	0.4539%, 1.7883%, 1.5112%
4403	4	20	1.9037%	3.6929%	0.7233%	2.5757%	0.3242%, 1.8831%, 1.2586%
9603	4	30	1.8217%	3.7451%	0.6689%	2.3609%	0.3639%, 2.0083%, 0.9685%
16,803	4	40	0.6372%	1.9704%	0.3920%	1.5497%	0.3269%, 1.8606%, 0.6983%
26,003	4	50	3.6993%	7.3510%	0.4207%	1.6748%	0.3127%, 1.5756%, 0.3559%

. We can observe that the errors drop at a favorable rate with an increase in k. On the other hand, an increase in m causes a much more pronounced increase in the number of trainable parameters (complexity of the network) but incurs, in comparison, only a moderate improvement of the accuracy level.

In Figure 3 we plot the effective potential W and the landscape function u.

3.5. Encoder–Decoder Network as a Reduced-Order Model for a Family of Landscape Functions

We now study the use of the sparse, U-Net-inspired [30], network architecture as a surrogate model for the function $\mathcal{L}:V\mapsto u$. In Figure 3 we show the landscape function u with periodic boundary conditions for the potential $V_{\omega }$ constructed as a lattice superposition of sixteen Gaussian bell functions $G\left(x\right)=\alpha exp(-|x_1-c_1{|}^2/k_1^2-|x_2-c_2{|}^2/k_2^2)$. The centers $c=(c_1,c_2)$ were chosen randomly inside each block of the $4\times 4$ uniform quadrilateral tessellation of $\mathsf{\Omega }$. The constants $\alpha ,k_i,i=1,2$ were chosen randomly and independently from intervals $[8,128]$ and $[1,1.5]$, respectively. To introduce local defects in the lattice, we have further randomly chosen three Gaussian bells and removed them from the potential. The choice of Gaussian bells to be removed was restricted, so that the boundary conditions were respected and that none of the erased bells were pairwise adjacent.

As the reduced order model for this family of problems, we have used the encoder–decoder fully convolutional neural network (FCNN) from Appendix C with 2,614,531 trainable parameters. This is a relatively small number of parameters in comparison with the typical convolutional neural network architectures with fully connected layers. The architecture of the neural network is shown in Figure A2.

To train the model we generated 98,400 potentials $V_{\omega }$ and then used FEniCS to compute the associated landscape functions $u_{\omega }$, $-\Delta u_{\omega }+V_{\omega }{u}_{\omega }=\mathbf{1}$. The domain of the Hamiltonaian was $\mathsf{\Omega }={[-10, 10]}^2$, with the periodic boundary conditions. We used the uniform quadrilateral discretization with the step size $h=20/50=0.4$ and $P_2$ elements to compute the training examples. To construct the reduced-order model, we projected (by interpolation) these $P_2$ functions onto the space of $P_1$ elements for the same mesh. After implementing the periodic boundary conditions we obtained exactly 2500 free nodes for this space of $P_1$ functions.

We denote the values of the potential V in those nodes as the vector $\overrightarrow{V}\in {\mathbb{R}}^{2,500}$ and we tacitly identify the vector $\overrightarrow{V}$ with the function $I_{P_1}V$. Let $I_{P_1}^{*}$ be the extension operator from ${\mathbb{R}}^{2500}$ to the space of continuous piece-wise linear functions. Then

$$\tilde{\mathcal{L}}\left(\overrightarrow{V}\right):=I_{P_1}\mathcal{L}\left(I_{P_1}^{*}{I}_{P_1}V\right)$$

defines the mapping $\tilde{\mathcal{L}}:{\mathbb{R}}^{2,500}\to {\mathbb{R}}^{2,500}$. We used the learning rate $\delta ={10}^{-4}$ for the first 100 epochs of the Adam optimizer and the learning rate $\delta ={10}^{-5}$ for a further 50 epochs. The activation function ${\rho }_{LU}$ was used to promote sparsity and the MSE loss function was used for the training. The batch size for the Adam algorithm was 1024. We implemented the certified surrogate modeling approach by combining the evaluation of the neural network with the error estimator $\tau$. In the case in which the residual measure $\tau$ for the function $\underline{u}=\mathcal{L}\left(I_{P_1}^{*}{I}_{P_1}\widehat{V}\right)$ is larger than the preset tolerance, we updated the surrogate model (neural network). To this end we solved in FEniCS the problem $-\Delta u+\widehat{V}u=\mathbf{1}$ and used the standard update algorithm for the convolutional neural network and the new training example.

We evaluated the performance of the neural network reduced-order model on a set of 200 testing potentials that were not used in the training of the network, see

Table 4. Validation of the encoder–decoder representation of the mapping $\mathcal{L}:V\mapsto u$ on a collection of test examples. Recall that the effective potential is defined as $W=1/u$.

Average ${\mathit{L}}^2$ error	1.7545%
Maximal $L^2$ error	2.9769%, example: 58
Average $H^1$ error	9.2233%
Maximal $H^1$ error	12.6765%, example: 65
Mean relative error in $1/W_{min,1}$	0.4887%
Maximal relative error in $1/W_{min,1}$	2.1402%, example: 70
The worst ten relative errors in $1/W_{min,1}$ (%)	2.1402, 1.5909, 1.5560, 1.4816, 1.4151, 1.4626, 1.3441, 1.3377, 1.3181, 1.3132

. The benchmarking comparison against the FEniCS solution is presented in Figure 4.

4. Discussion

According to the authors of [45], deep learning approaches to dealing with partial differential equations fall into the following three categories: (a) Rayleigh–Ritz approximations, (b) Galerkin approximations and (c) least squares approximations. We have considered a hybrid approach that combines robust stochastic optimization of overparametrized networks based on the Rayleigh–Ritz approach with standard residual based error estimates, which together yield a hybrid approximation method. We were particularly influenced by the review in [45] and the Deep Ritz algorithm as described in [20].

In the example from Table 3 we further computed a piece-wise cubic and piece-wise quadratic approximation of the landscape function using the standard finite element method and an approximation of the landscape function using a variant of the Deep Ritz method. We measured the error of the $P_2$ and the VPINN approximation against the $P_3$ benchmark solution. The relative $L^2$ error of the piece-wise quadratic approximation was computed to be $0.36\%$, whereas the relative error of the VPINN approximation with 1203 free parameters was computed to be $1.21\%$. Note, however, that the piece-wise quadratic approximation required the training of 10,000 parameters. This indicates that the neural network achieves a considerable data compression when compared with a piece-wise polynomial approximation. The situation is even more interesting in 1D. There we compared the ground state approximation by the Chebyshev series, as implemented in chebfun. We needed 149 terms in the Chebyshev expansion to reach the order of machine precision. On the other hand, the Adam optimizer was able to find a realization of the dense neural network with VPINN architecture ${\overrightarrow{N}}_{\mathtt{DenseNet}}=(1, 2, 2, 2)$ from Appendix A. This architecture only has 30 trainable parameters and it achieves a relative distance (in the $L^2$ sense) of $1.05\%$ to the benchmark chebfun solution. The relative error in the ground state energy is only $0.1\%$, see Figure 5.

The integrals needed to approximate the energy functional were computed using Gaussian quadrature rules. Unfortunately, this approach does not yield stable methods in higher-dimensional problems. The reason is in the fact that sparse grid quadratures (e.g., [40]) also have nodes with negative weights and this was observed to cause severe numerical instability. An approach based on the quasi-Monte Carlo integration, which utilizes low-discrepancy sequences of integration nodes and has only positive weights (see [39]) yielded an efficient and stable method in higher-dimensional situations. Furthermore, since realizations of neural networks are frequently functions with many local extrema, computing their integrals needs to be handled with care. This is particularly relevant when enforcing discretizations of physical or normalization constraints by penalization. We point out that the scalability of sparse-grid integration schemes in this respect was not satisfactory.

Another promising technique for obtaining data-sparse compressed approximation of the solutions of partial differential equations is based on the concept of tensor networks, also known as matrix product states or tensor train decompositions [9]. This approach has been successfully converted into numerical approximation algorithms such as the quantized tensor train decompositions [8,46]. The scaling robust performance of this approach has been demonstrated on a class of multi-scale model problems in [46]. However, the numerical methods still have to be tailor-made for the chosen problems. On the other hand, there are many freely available robust and highly developed libraries for working with deep neural networks. This is the reason for our choice of the discretization method.

5. Conclusions

We have presented two types of neural network architectures. First, a dense deep network of the DenseNet type was used as a compressed approximation of the ground state and the landscape function of the problems under consideration. Remarkably, it achieved high accuracy and a good compression rate even when empirically compared with a Chebyshev expansion in 1D. Even though we managed to tackle problems in ${\mathbb{R}}^n$, this concept struggled to yield scalable numerical methods. We then took another approach and considered a problem of approximating a mapping $\mathcal{L}:{\mathbb{R}}^{2500}\to {\mathbb{R}}^{2500}$, which connects a mesh sample of a potential with the associated landscape function. A fully convolutional neural network architecture with the ReLU activation function, to further promote sparsity, turned out to be expressive enough to deal with this family of problems to a satisfactory level of accuracy (empirically measured on the test set). We have also seen that a hybrid approach—one that combines the expressivity of the set of neural network realizations with the standard error indicators—has a potential to lead to robust approximation methods. We have observed that it is particularly challenging to turn physical constraints—which are continuous—into their discrete realizations, which can be used to filter out, e.g., by judiciously applied penalization, the nonphysical neural network realizations from the set of all realizations of a given architecture. How to turn this into a robust mesh-less and scalable method for dealing with equations of mathematical physics will be a topic of further research.