# Solving Partial Differential Equations Using Deep Learning and Physical Constraints

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Methodology

#### 2.1. Artificial Neural Networks

#### 2.2. Physics-Informed Neural Networks

## 3. Experiments and Results

#### 3.1. Wave Equation

#### 3.2. KdV-Burgers Equation

#### 3.3. Two-Soliton Solution of the Korteweg-De Vries Equation

## 4. Discussions

- (1)
- The most important task of physics-informed neural networks is to introduce a reasonable regularization of physical information. The use of physical information allows neural networks to better learn the solutions of partial differential equations from fewer observations. In this study, the physical information regularization is implemented through the automatic differentiation of the TensorFlow framework, which may not be applicable in many practical problems. Therefore, we need to develop more general differential methods and expand more methods for introducing physical information so that physics-informed neural networks can be better applied to real-world problems.
- (2)
- Regularization has an important role in physics-informed neural networks. Similar to the previous related studies, the regularization in this study takes the form of the L2 norm. However, when considering the advantages of different norms, such as the ability of L1 norm to resist anomalous data interference, in the next study, we will adopt different forms of regularization of physical information such as L1 norm, to further improve the theory and methods related to physics-informed neural networks.
- (3)
- In this study, the training data used to train the physics-informed neural network are randomly selected in the space-time domain, thus the physics-informed neural network does not need to consider the discretization of partial differential equations and can learn the solutions of partial differential equations from a small amount of data. It is well known that popular computational fluid dynamics methods require consideration of the discretization of equations, such as finite difference methods. In practice, many engineering applications also need to consider the discretization of partial differential equations, for example, various numerical weather prediction models have made discretization schemes as an important part of their research. Physics-informed neural networks are well suited to solve this problem and, thus, this approach may have important implications for the development of computational fluid dynamics and even scientific computing.
- (4)
- Although the method used in this paper has many advantages, such as not having to consider the discretization of PDEs. However, the method also faces many problems, such as the neural network for solving PDEs relies heavily on training data, which often requires more training time when the quality of the training data is poor. Therefore, it is also important to investigate how to construct high-quality training datasets to reduce the training time.
- (5)
- In this study, we focus on solving PDEs by training physics-informed neural networks, which is a supervised learning task. Currently, several researchers have used unlabeled data to train physics-constrained deep learning models for high-dimensional problems and have quantified the uncertainty of the predictions [68]. This inspires us to further improve our neural network, so that it can be trained using unlabeled data and give a probabilistic interpretation of the prediction results [69]. Besides, this paper studies low-dimensional problems, whereas, for high-dimensional problems, model reduction [70] is also an important issue to consider when constructing a neural network model.
- (6)
- In this paper, we study one-dimensional partial differential equations, but the method can be applied to multi-dimensional problems. Currently, we are attempting to apply the method to the simulation of three-dimensional atmospheric equations for improving existing numerical weather prediction models. Besides, in the field of engineering, complex PDE systems of field coupled nature, as in Fluid-Structure Interaction, are of great research value and they are widely used in the aerospace industry, nuclear engineering, etc. [71], so we will also study such complex PDEs in the future.
- (7)
- So far, there have been many promising applications of neural networks in computational engineering. For example, one very interesting work is that neural networks have been used to construct constitutive laws as a surrogate model replacing the two-scale computational approaches [72,73]. These valuable works will guide us in further exploring the applications of neural networks in scientific computing.

## 5. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## Abbreviations

PDEs | Partial Differential Equations |

ANN | Artificial Neural Networks |

FNN | Feedforward Neural Network |

PINN | Physics Informed Neural Network |

FEM | Finite Element Method |

FDM | Finite Difference Method |

FVM | Finite Volume Method |

AD | Automatic Differentiation |

SGD | Stochastic Gradient Descent |

Adam | Adaptive Moment Estimation |

L-BFGS | Limited-memory BFGS |

KdV equation | Korteweg-de Vries equation |

## References

- Folland, G.B. Introduction to Partial Differential Equations; Princeton University Press: Princeton, NJ, USA, 1995; Volume 102. [Google Scholar]
- Petrovsky, I.G. Lectures on Partial Differential Equations; Courier Corporation: North Chelmsford, MA, USA, 2012. [Google Scholar]
- Courant, R.; Hilbert, D. Methods of Mathematical Physics: Partial Differential Equations; John Wiley & Sons: Hoboken, NJ, USA, 2008. [Google Scholar]
- Farlow, S.J. Partial Differential Equations for Scientists and Engineers; Courier Corporation: North Chelmsford, MA, USA, 1993. [Google Scholar]
- Zauderer, E. Partial Differential Equations of Applied Mathematics; John Wiley & Sons: Hoboken, NJ, USA, 2011; Volume 71. [Google Scholar]
- Churchfield, M.J.; Lee, S.; Michalakes, J.; Moriarty, P.J. A numerical study of the effects of atmospheric and wake turbulence on wind turbine dynamics. J. Turbul.
**2012**, 13, N14. [Google Scholar] [CrossRef] - Müller, E.H.; Scheichl, R. Massively parallel solvers for elliptic partial differential equations in numerical weather and climate prediction. Q. J. R. Meteorol. Soc.
**2014**, 140, 2608–2624. [Google Scholar] [CrossRef][Green Version] - Tröltzsch, F. Optimal Control of Partial Differential Equations: Theory, Methods, and Applications; American Mathematical Society: Providence, RI, USA, 2010; Volume 112. [Google Scholar]
- Ames, W.F. Numerical Methods for Partial Differential Equations; Academic Press: Cambridge, MA, USA, 2014. [Google Scholar]
- Quarteroni, A.; Valli, A. Numerical Approximation of Partial Differential Equations; Springer: Berlin/Heidelberg, Germany, 2008; Volume 23. [Google Scholar]
- Yin, S.; Ding, S.X.; Xie, X.; Luo, H. A review on basic data-driven approaches for industrial process monitoring. IEEE Trans. Ind. Electron.
**2014**, 61, 6418–6428. [Google Scholar] [CrossRef] - Bai, Z.; Brunton, S.L.; Brunton, B.W.; Kutz, J.N.; Kaiser, E.; Spohn, A.; Noack, B.R. Data-driven methods in fluid dynamics: Sparse classification from experimental data. In Whither Turbulence and Big Data in the 21st Century? Springer: Berlin/Heidelberg, Germany, 2017; pp. 323–342. [Google Scholar]
- Goodfellow, I.; Bengio, Y.; Courville, A. Deep Learning; MIT Press: Cambridge, MA, USA, 2016. [Google Scholar]
- LeCun, Y.; Bengio, Y.; Hinton, G. Deep learning. Nature
**2015**, 521, 436–444. [Google Scholar] [CrossRef] [PubMed] - Deng, L.; Yu, D. Deep learning: Methods and applications. Found. Trends Signal Process.
**2014**, 7, 197–387. [Google Scholar] [CrossRef][Green Version] - Li, S.; Song, W.; Fang, L.; Chen, Y.; Ghamisi, P.; Benediktsson, J.A. Deep learning for hyperspectral image classification: An overview. IEEE Trans. Geosci. Remote. Sens.
**2019**, 57, 6690–6709. [Google Scholar] [CrossRef][Green Version] - Goldberg, Y. A primer on neural network models for natural language processing. J. Artif. Intell. Res.
**2016**, 57, 345–420. [Google Scholar] [CrossRef][Green Version] - Helbing, G.; Ritter, M. Deep Learning for fault detection in wind turbines. Renew. Sustain. Energy Rev.
**2018**, 98, 189–198. [Google Scholar] [CrossRef] - Lu, Y.; Lu, J. A Universal Approximation Theorem of Deep Neural Networks for Expressing Distributions. arXiv
**2020**, arXiv:2004.08867. [Google Scholar] - Hornik, K.; Stinchcombe, M.; White, H. Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks. Neural Networks
**1990**, 3, 551–560. [Google Scholar] [CrossRef] - Lagaris, I.E.; Likas, A.; Fotiadis, D.I. Artificial neural networks for solving ordinary and partial differential equations. IEEE Trans. Neural Networks
**1998**, 9, 987–1000. [Google Scholar] [CrossRef] [PubMed][Green Version] - Göküzüm, F.S.; Nguyen, L.T.K.; Keip, M.A. An Artificial Neural Network Based Solution Scheme for Periodic Computational Homogenization of Electrostatic Problems. Math. Comput. Appl.
**2019**, 24, 40. [Google Scholar] [CrossRef][Green Version] - Nguyen-Thanh, V.M.; Zhuang, X.; Rabczuk, T. A deep energy method for finite deformation hyperelasticity. Eur. J. -Mech.-A/Solids
**2020**, 80, 103874. [Google Scholar] [CrossRef] - Bar, L.; Sochen, N. Unsupervised deep learning algorithm for PDE-based forward and inverse problems. arXiv
**2019**, arXiv:1904.05417. [Google Scholar] - Freund, J.B.; MacArt, J.F.; Sirignano, J. DPM: A deep learning PDE augmentation method (with application to large-eddy simulation). arXiv
**2019**, arXiv:1911.09145. [Google Scholar] - Wu, K.; Xiu, D. Data-driven deep learning of partial differential equations in modal space. J. Comput. Phys.
**2020**, 408, 109307. [Google Scholar] [CrossRef][Green Version] - Khoo, Y.; Lu, J.; Ying, L. Solving parametric PDE problems with artificial neural networks. arXiv
**2017**, arXiv:1707.03351. [Google Scholar] - Huang, X. Deep neural networks for waves assisted by the Wiener–Hopf method. Proc. R. Soc.
**2020**, 476, 20190846. [Google Scholar] [CrossRef][Green Version] - Raissi, M.; Perdikaris, P.; Karniadakis, G.E. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations. J. Comput. Phys.
**2019**, 378, 686–707. [Google Scholar] [CrossRef] - Raissi, M.; Perdikaris, P.; Karniadakis, G.E. Physics Informed Deep Learning (Part I): Data-driven Solutions of Nonlinear Partial Differential Equations. arXiv
**2017**, arXiv:1711.10561. [Google Scholar] - Raissi, M.; Perdikaris, P.; Karniadakis, G.E. Physics Informed Deep Learning (Part II): Data-driven Discovery of Nonlinear Partial Differential Equations. arXiv
**2017**, arXiv:1711.10566. [Google Scholar] - Chen, X.; Duan, J.; Karniadakis, G.E. Learning and meta-learning of stochastic advection-diffusion-reaction systems from sparse measurements. arXiv
**2019**, arXiv:1910.09098. [Google Scholar] - He, Q.; Barajas-Solano, D.; Tartakovsky, G.; Tartakovsky, A.M. Physics-informed neural networks for multiphysics data assimilation with application to subsurface transport. Adv. Water Resour.
**2020**, 141, 103610. [Google Scholar] [CrossRef] - Tartakovsky, A.; Marrero, C.O.; Perdikaris, P.; Tartakovsky, G.; Barajas-Solano, D. Physics-Informed Deep Neural Networks for Learning Parameters and Constitutive Relationships in Subsurface Flow Problems. Water Resour. Res.
**2020**, 56, e2019WR026731. [Google Scholar] [CrossRef] - Kadeethum, T.; Jørgensen, T.M.; Nick, H.M. Physics-informed neural networks for solving nonlinear diffusivity and Biot’s equations. PLoS ONE
**2020**, 15, e0232683. [Google Scholar] [CrossRef] [PubMed] - Kissas, G.; Yang, Y.; Hwuang, E.; Witschey, W.R.; Detre, J.A.; Perdikaris, P. Machine learning in cardiovascular flows modeling: Predicting arterial blood pressure from non-invasive 4D flow MRI data using physics-informed neural networks. Comput. Methods Appl. Mech. Eng.
**2020**, 358, 112623. [Google Scholar] [CrossRef] - Jagtap, A.D.; Kawaguchi, K.; Karniadakis, G.E. Adaptive activation functions accelerate convergence in deep and physics-informed neural networks. J. Comput. Phys.
**2020**, 404, 109136. [Google Scholar] [CrossRef][Green Version] - Shanmuganathan, S. Artificial neural network modelling: An introduction. In Artificial Neural Network Modelling; Springer: Berlin/Heidelberg, Germany, 2016; pp. 1–14. [Google Scholar]
- Nielsen, M.A. Neural Networks and Deep Learning; Determination Press: San Francisco, CA, USA, 2015; Volume 2018. [Google Scholar]
- Svozil, D.; Kvasnicka, V.; Pospichal, J. Introduction to multi-layer feed-forward neural networks. Chemom. Intell. Lab. Syst.
**1997**, 39, 43–62. [Google Scholar] [CrossRef] - Li, J.; Cheng, J.H.; Shi, J.Y.; Huang, F. Brief introduction of back propagation (BP) neural network algorithm and its improvement. In Advances in Computer Science and Information Engineering; Springer: Berlin/Heidelberg, Germany, 2012; pp. 553–558. [Google Scholar]
- Zhang, D.; Guo, L.; Karniadakis, G.E. Learning in modal space: Solving time-dependent stochastic PDEs using physics-informed neural networks. SIAM J. Sci. Comput.
**2020**, 42, A639–A665. [Google Scholar] [CrossRef] - Tipireddy, R.; Perdikaris, P.; Stinis, P.; Tartakovsky, A. A comparative study of physics-informed neural network models for learning unknown dynamics and constitutive relations. arXiv
**2019**, arXiv:1904.04058. [Google Scholar] - Abadi, M.; Barham, P.; Chen, J.; Chen, Z.; Davis, A.; Dean, J.; Devin, M.; Ghemawat, S.; Irving, G.; Isard, M.; et al. Tensorflow: A system for large-scale machine learning. In Proceedings of the 12th {USENIX} symposium on operating systems design and implementation ({OSDI} 16), Savannah, GA, USA, 2–4 November 2016; pp. 265–283. [Google Scholar]
- Paszke, A.; Gross, S.; Chintala, S.; Chanan, G.; Yang, E.; DeVito, Z.; Lin, Z.; Desmaison, A.; Antiga, L.; Lerer, A. Automatic differentiation in pytorch. In Proceedings of the NIPS 2017 Autodiff Workshop, Long Beach, CA, USA, 9 December 2017. [Google Scholar]
- Bengio, Y. Gradient-based optimization of hyperparameters. Neural Comput.
**2000**, 12, 1889–1900. [Google Scholar] [CrossRef] [PubMed] - Kylasa, S.; Roosta, F.; Mahoney, M.W.; Grama, A. GPU accelerated sub-sampled newton’s method for convex classification problems. In Proceedings of the 2019 SIAM International Conference on Data Mining, Calgary, AB, Canada, 2–4 May 2019; pp. 702–710. [Google Scholar]
- Richardson, A. Seismic full-waveform inversion using deep learning tools and techniques. arXiv
**2018**, arXiv:1801.07232. [Google Scholar] - Le, Q.V.; Ngiam, J.; Coates, A.; Lahiri, A.; Prochnow, B.; Ng, A.Y. On optimization methods for deep learning. In Proceedings of the 28th International Conference on Machine Learning, ICML, Bellevue, WA, USA, 28 June–2 July 2011. [Google Scholar]
- Lu, L.; Meng, X.; Mao, Z.; Karniadakis, G.E. DeepXDE: A deep learning library for solving differential equations. arXiv
**2019**, arXiv:1907.04502. [Google Scholar] - Li, J.; Feng, Z.; Schuster, G. Wave-equation dispersion inversion. Geophys. J. Int.
**2017**, 208, 1567–1578. [Google Scholar] [CrossRef] - Gu, J.; Zhang, Y.; Dong, H. Dynamic behaviors of interaction solutions of (3+ 1)-dimensional Shallow Water wave equation. Comput. Math. Appl.
**2018**, 76, 1408–1419. [Google Scholar] [CrossRef] - Kim, D. A Modified PML Acoustic Wave Equation. Symmetry
**2019**, 11, 177. [Google Scholar] [CrossRef][Green Version] - Raissi, M.; Perdikaris, P.; Karniadakis, G.E. Numerical Gaussian processes for time-dependent and non-linear partial differential equations. arXiv
**2017**, arXiv:1703.10230. [Google Scholar] - Hanin, B.; Rolnick, D. How to start training: The effect of initialization and architecture. In Advances in Neural Information Processing Systems; Curran Associates Inc.: Red Hook, NY, USA, 2018; pp. 571–581. [Google Scholar]
- Samokhin, A. Nonlinear waves in layered media: Solutions of the KdV–Burgers equation. J. Geom. Phys.
**2018**, 130, 33–39. [Google Scholar] [CrossRef] - Zhang, W.G.; Li, W.X.; Deng, S.E.; Li, X. Asymptotic Stability of Monotone Decreasing Kink Profile Solitary Wave Solutions for Generalized KdV-Burgers Equation. Acta Math. Appl. Sin. Engl. Ser.
**2019**, 35, 475–490. [Google Scholar] [CrossRef] - Samokhin, A. On nonlinear superposition of the KdV–Burgers shock waves and the behavior of solitons in a layered medium. Differ. Geom. Appl.
**2017**, 54, 91–99. [Google Scholar] [CrossRef] - Ahmad, H.; Seadawy, A.R.; Khan, T.A. Numerical solution of Korteweg–de Vries-Burgers equation by the modified variational iteration algorithm-II arising in shallow water waves. Phys. Scr.
**2020**, 95, 045210. [Google Scholar] [CrossRef] - Seydaoğlu, M.; Erdoğan, U.; Öziş, T. Numerical solution of Burgers’ equation with high order splitting methods. J. Comput. Appl. Math.
**2016**, 291, 410–421. [Google Scholar] [CrossRef] - Khalique, C.M.; Mhlanga, I.E. Travelling waves and conservation laws of a (2+ 1)-dimensional coupling system with Korteweg-de Vries equation. Appl. Math. Nonlinear Sci.
**2018**, 3, 241–254. [Google Scholar] [CrossRef][Green Version] - Wang, Y.; Navon, I.M.; Wang, X.; Cheng, Y. 2D Burgers equation with large Reynolds number using POD/DEIM and calibration. Int. J. Numer. Methods Fluids
**2016**, 82, 909–931. [Google Scholar] [CrossRef] - Wu, J.; Geng, X. Inverse scattering transform and soliton classification of the coupled modified Korteweg-de Vries equation. Commun. Nonlinear Sci. Numer. Simul.
**2017**, 53, 83–93. [Google Scholar] [CrossRef] - Khusnutdinova, K.R.; Stepanyants, Y.; Tranter, M.R. Soliton solutions to the fifth-order Korteweg–de Vries equation and their applications to surface and internal water waves. Phys. Fluids
**2018**, 30, 022104. [Google Scholar] [CrossRef][Green Version] - Driscoll, T.A.; Hale, N.; Trefethen, L.N. Chebfun Guide; Pafnuty Publications: Oxford, UK, 2014. [Google Scholar]
- Nguyen, L.T.K. Modified homogeneous balance method: Applications and new solutions. Chaos Solitons Fractals
**2015**, 73, 148–155. [Google Scholar] [CrossRef] - Nguyen, L.T.K. Soliton solution of good Boussinesq equation. Vietnam. J. Math.
**2016**, 44, 375–385. [Google Scholar] [CrossRef] - Zhu, Y.; Zabaras, N.; Koutsourelakis, P.S.; Perdikaris, P. Physics-Constrained Deep Learning for High-dimensional Surrogate Modeling and Uncertainty Quantification without Labeled Data. J. Comput. Phys.
**2019**, 394, 56–81. [Google Scholar] [CrossRef][Green Version] - Murphy, K.P. Machine Learning: A Probabilistic Perspective; MIT Press: Cambridge, MA, USA, 2012. [Google Scholar]
- Chinesta, F.; Ladeveze, P.; Cueto, E. A Short Review on Model Order Reduction Based on Proper Generalized Decomposition. Arch. Comput. Methods Eng.
**2011**, 18, 395–404. [Google Scholar] [CrossRef][Green Version] - Ohayon, R.; Schotté, J.S. Fluid–Structure Interaction Problems. In Encyclopedia of Computational Mechanics, 2nd ed.; American Cancer Society: Atlanta, GA, USA, 2017; pp. 1–12. [Google Scholar]
- Nguyen-Thanh, V.M.; Nguyen, L.T.K.; Rabczuk, T.; Zhuang, X. A surrogate model for computational homogenization of elastostatics at finite strain using HDMR-based neural network. Int. J. Numer. Methods Eng.
**2020**. [Google Scholar] [CrossRef] - Le, B.; Yvonnet, J.; He, Q.C. Computational homogenization of nonlinear elastic materials using neural networks. Int. J. Numer. Methods Eng.
**2015**, 104, 1061–1084. [Google Scholar] [CrossRef]

**Figure 1.**A structural diagram of a feedforward neural network (FNN), which consists of an input layer, one or more hidden layers, and an output layer, each containing one or more artificial neurons.

**Figure 2.**The schematic of physics-informed neural network (PINN) for solving partial differential equations.

**Figure 4.**Comparison of the prediction given by physics-informed neural networks with the exact solution.

**Figure 5.**Solution of the Korteweg-de Vries (KdV)–Burgers equation given by physics-informed neural networks.

**Figure 6.**Comparison of the prediction given by physics-informed neural networks with the exact solution.

**Figure 8.**Comparison of the prediction given by physics-informed neural networks with the exact solution.

© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Guo, Y.; Cao, X.; Liu, B.; Gao, M.
Solving Partial Differential Equations Using Deep Learning and Physical Constraints. *Appl. Sci.* **2020**, *10*, 5917.
https://doi.org/10.3390/app10175917

**AMA Style**

Guo Y, Cao X, Liu B, Gao M.
Solving Partial Differential Equations Using Deep Learning and Physical Constraints. *Applied Sciences*. 2020; 10(17):5917.
https://doi.org/10.3390/app10175917

**Chicago/Turabian Style**

Guo, Yanan, Xiaoqun Cao, Bainian Liu, and Mei Gao.
2020. "Solving Partial Differential Equations Using Deep Learning and Physical Constraints" *Applied Sciences* 10, no. 17: 5917.
https://doi.org/10.3390/app10175917