Non-intrusive inference reduced order model for fluids using deep multistep neural network

Non-intrusive inference reduced order model for fluids using deep multistep neural network Xuping Xie 1,‡*, Guannan Zhang 1,‡ and Clayton G. Webster 1,2,‡ 1 Computation and Applied Mathematics, Oak Ridge National Laboratory; xiex@ornlg.gov; zhangg@ornl.gov; 2 Department of Mathematics; University of Tennessee, Knoxville; cwebst13@utk.edu * Correspondence: xiex@ornl.gov ‡ These authors contributed equally to this work.


Introduction
The full order model (FOM) of realistic engineering applications in fluid dynamics often represents a large scale dynamic system. High-fidelity CFD (computational fluid dynamics) simulations of the FOM are so computationally expensive that they put a heavy burden on the computational resources despite the available CFD software and supercomputers with thousands of cores. Consequently, the use of FOM for such simulations is often impractical and prohibitive for time-critical applications such as system identification, flow control, design optimization.
The reduced order modeling in fluid dynamics is to construct an accurate low-dimensional approximation to the full system with orders of magnitude reduction in computational cost. The first usage of reduced order model (ROM) in fluid dynamics was by (Lumley 1967) for studying the intensity of turbulence and coherent structures. Many recent successful applications of ROM in fluid problems can be found in Copyright c 2019, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. (Noack, Morzynski, and Tadmor 2011;Obinata and Anderson 2012;Carlberg et al. 2013;Rowley and Dawson 2017;Amsallem and Farhat 2012;San and Maulik 2018a) The Galerkin projection based reduced order model (GP-ROM) is one of the most popular methods that has been widely used in practice. The GP-ROM, in an offline stage, first constructs a reduced space and then uses the Galerkin projection of the FOM operator to obtain a low-dimensional nonlinear dynamical system, i.e., ROM dynamics. The reduced space is often generated by proper orthogonal decomposition (POD), also known as principal component analysis. In an online stage, the obtained reduced dynamics can be used to approximate the full system efficiently for various applications, such as long-term prediction, flow control. However, the projection step requires that the full model operators have to be available in order to obtain the ROM dynamics. This limits the applicability of projection based model reduction in situations where the full model is unknown (Xiao et al. 2015;Peherstorfer and Willcox 2016). More importantly, the computational cost of assembling the reduced operators -tensors from the projection of FOM operator -scales with the large dimension of the underlying high-dimensional of FOM. For this reason, the GP-ROM are efficient for problems where the reduced operators must be constructed only once.
On the other hand, the GP-ROM generates inaccurate approximations for highly non-stationary (nonlinear) fluids, e.g., turbulence. In the literature, a common explanation for this failure is that the Galerkin projection does not preserve the stability properties from the full model. A deeper reason is that the low-dimensional space used in the Galerkin projection cannot resolve the nonlinear interaction of the fluid system (Noack et al. 2016;Loiseau and Brunton 2018). Consequently, resulting in a projection based stability error which makes GP-ROM fail in nonlinear fluid applications, e.g., see (Balajewicz, Dowell, and Noack 2013;Ballarin et al. 2015;Carlberg, Barone, and Antil 2015;Xie et al. 2018) Related Work Closure modeling. Numerous stabilization strategies have been devised to address the instability problem, known as closure modeling. The fundamental idea of closure modeling is to model the lost information from the lowdimensional space since it is generated through POD truncation. This truncation is to keep the first few POD modes that extract the most dominant structure of the full system and discard the rest modes. Closure models generally can be categorized into two common approaches. One is physically model the effect of the discarded POD mode by adding artificial viscosity to the reduced system (Balajewicz, Dowell, and Noack 2013;Ballarin et al. 2015;Protas, Noack, andÖsth 2015;Amsallem and Farhat 2012). Another approach is to mathematically model the ROM dynamics by solving a related optimization problem (Carlberg, Barone, and Antil 2015;Xie et al. 2018). The most recent development for closure modeling is to apply the neural network to approximate the lost information (San and Maulik 2018b).
Differential equations learning. Bridging numerical differential equations and deep neural networks have gained enormous attention recently. Specifically, Chang et al. (2017) proposed dynamical system viewing of residual networks (ResNet). Lu et al. (2017) the first time introduced the linear multistep network architecture to analyze ResNet on classification task. Sparse regression, Gaussian process, multistep neural networks have been applied for dynamic system learning Rudy et al. 2017;Raissi and Karniadakis 2018;Raissi, Perdikaris, and Karniadakis 2017). More recently, ordinary differential equation network (ODE-net) was introduced for supervised learning (Chen et al. 2018).

Our approach
In this paper, we propose a novel non-intrusive learning reduced order model framework for fluid dynamic system. The new framework provides a general concept of learning the optimal reduced dynamic system from the data. Inspired by the successful development in learning differential equations with deep networks, we apply the linear multistep neural network (LMNet) to learn the reduced order model (LMNet-ROM). Unlike closure modeling and the existed non-intrusive model reduction methods, in this work, we focus on a different perspective. First, we do not use Galerkin projection and model the closure problem. The optimal reduced dynamic system that can address the instability issue is learned for a given supervised learning task. Secondly, the new model does not approximate reduced operators whereas other non-intrusive models use interpolation or regression method to infer reduced operators (Peherstorfer and Willcox 2016;Xiao et al. 2015). Moreover, our viewpoint easily enables us to answer the common question -what is the best ROM dynamic system to approximate the full system for a given low-dimensional subspace? We demonstrate the performance of the new LMNet-ROM is better than GP-ROM in full order model approximation and long-term prediction. The main contributions can be summarized as follows: • A novel non-intrusive learning reduced order model framework for fluid dynamics, which is applicable to general nonlinear dynamical systems with sophisticated legacy codes.
• Our framework overcomes the instability issue of the projection based model reduction, and provides accurate approximation and long-term prediction of the full system. • The learning process of our approach is more computationally efficient than the construction of reduced operators in the classic projection based methods.

Reduced Order Modeling
In this section, we present the Galerkin projection based reduced order modeling framework in fluid dynamic system. The classical Navier-Stokes equations (NSE) are often used as a mathematical model in fluid dynamics, where u is the velocity, p the pressure, and Re the Reynolds number. We use the initial condition u(x, 0) = u 0 (x) and (for simplicity) homogeneous Dirichlet boundary conditions: u(x, t) = 0. For convenience, we use ∂u ∂t = f(u, Re) as the general notation for the NSE in the rest of the paper.
Reduced Space. Proper orthogonal decomposition (POD) is the dimension reduction method that we used to generate the reduced space. It starts with the data matrix, , collected by the numerical solutions or experimental observations of the full system (1) at s+1 different time instances. The POD method seeks a low-dimensional space X r that approximates the data (U ) optimally with respect to the L 2 -norm. It formulates the following eigenvalue problem: where d is the rank of the data matrix U U T . λ i and ϕ i are the eigenvalues and POD basis, respectively. The reduced space is given after the truncation as Φ r := {ϕ 1 , · · · , ϕ r } ∈ R D h ×r . Galerkin Projection-ROM (GP-ROM). For a given space Φ r , the GP-ROM finds the approximation of the velocity field spanned by the low-dimensional space, where {a j (t)} r j=1 are the sought time-varying coefficients. The GP-ROM can be obtained by projecting the FOM onto the POD space: ∀ i = 1, . . . , r, Here, (·, ·) is the L 2 inner product. The solution of GP-ROM can be determined by the following nonlinear dynamic system:ȧ where L and N are ROM operators that can be obtained by projection Note that the reduced system (6) efficiently approximates the full model of NSE as the dimension r is generally very small (∼ O(10)) compared to the high dimension of u (D h ∼ O(≥ 10 5 )). ROM closure models. The ROM closure models can be generally written as the following dynamic system: τ is an artificial term that model the effect of the discarded POD modes using various approaches e.g., see (Parish and Duraisamy 2016;Gouasmi, Parish, and Duraisamy 2017;Xie et al. 2018;San and Maulik 2018b). We note that the dynamics of closure model (8) is more accurate than GP-ROM dynamics (6) in approximation of the model. The challenge of closure modeling is that the τ is unknown, i.e., no explicit formula. Therefore, closure models are empirical modification of the GP-ROM from physical or mathematical perspective.

Learning Reduced Order Model
In this section, we present the architecture of learning the reduced order model from deep neural networks. In contrast to the standard GP-ROM framework, we learn the reduced dynamical system from the data without intrusively using ROM operators (e.g., L, N )

Optimal ROM dynamics
We consider the low dimensional ROM dynamic system as a general function,ȧ = f r (a).
We claim that this function f r has a general representation of the ROM dynamics including system (6) and (8). Our goal is to learn the ROM dynamics (9) in a given set of temporal data and return a closed form model that can be used to accurately approximate and predict the full system. For a given data-set of snapshot solutions of NSE (1), U = u 0 , ..., u s ∈ R D h ×s+1 , at time steps t 0 , ..., t s . The best approximation of snapshots data by the POD space is given by, The reduced dynamics (9) is to find coefficients a such that, It indicates that the optimal solution from the r-dimensional ROM dynamic system is given by the full model data, b j , such thatḃ This provides a framework that learning the ROM dynamics from the data-set B = [b 0 , ..., b s ] ∈ R r×(s+1) , i.e., timevarying coefficients of FOM data. The training data-set can be computed by the following, The formula is derived by the L 2 projection of data U to the low-dimensional space Φ r . W ∈ R D h ×D h is the weight matrix of the L 2 inner product, we use finite element weights in this work.

Linear Multistep Network (LMNet)
Motivated by differential equation learning, we adopt the linear multistep network architecture (Lu et al. 2017;Raissi, Perdikaris, and Karniadakis 2018) to construct a structured nonlinear regression model that can learn the reduced dynamics. In this work, we only consider the implicit multistep method, Adams-Moulton (AM) scheme (Ascher and Petzold 1998) as it has better stability property. The K-step AM method is defined as follows: (α i a n−i + β i ∆tf r (a n−i )) + β 0 ∆tf r (a n ) (13) We discretize the ROM dynamic system (9) by using AM scheme (13) with a neural network. The parameters of this neural network can be learned by minimizing the mean squared error loss function: Where S is the total number of time instance in the ROM dynamic system. L n is the local truncation error from the Taylor expansion of K-step AM method (13), α i a n−i + ∆tβ i f r (a n−1 ).
One advantage of this nonlinear regression is that we do not have to approximate the temporal gradients (Peherstorfer and Willcox 2016;Rudy et al. 2017) since the time derivatives are discretized by the AM method.

LMNet-ROM
We use the trained neural network as the ROM dynamic system (9) to approximate the full system of NSE. We emphasize that the novelty of our approach is the non-intrusively learning of the reduced system, whereas GP-ROM and closure models require the use of FOM operators. Fig.1 is the flowchart of the LMNet-ROM and GP-ROM framework. We outline the algorithm of the framework in the following: Algorithm 1 Linear Multistep Network Reduced Order Model Learning (LMNet-ROM) 1: Compute the reduced POD space from the data of NSE by (3) 2: Compute the training dataset B via (12) 3: Train the neural network using loss function (14) 4: The LMNet-ROM for NSE is obatined from the trained low-dimensional dynamic system: We claim that the learned reduced dynamics (16) has a better approximation to the full model than system (7) and (8) since it is learned optimally from the FOM data. The new framework, see Fig.1, only requires input data from a system Figure 1: Flowchart of projection based model reduction and the new non-intrusive learning reduced order modeling framework and does not use any FOM operator, which can be generally applied to reduced order modeling of any fluid dynamical system. The main offline computational cost of the LMNet-ROM is training the neural network whereas the GP-ROM is the construction of ROM operators in (7). In the numerical experiment, we show that the offline computation of our model is faster than the GP-ROM.

Numerical Experiment
In this section, we present the preliminary numerical results to demonstrate the advantages of our new model. The test case is a 2D channel flow past a circular cylinder at a Re = 100. It is a benchmark problem that has been used as a numerical test in fluid dynamics, see (Schäefer and Turek 1996;Brunton et al. 2014;Mohebujjaman et al. 2017;Xie et al. 2018).

Implementation Details
The domain is a 2.2 × 0.41 rectangular channel with a radius=0.05 cylinder, centered at (0.2, 0.2), see Figure 2.
The velocity snapshots of NSE (1) are generated by finite element method with meshsize approximately 103K which gives a fully resolved solution. The lift and drag computation agree well with results from references in (Caiazzo et al. 2014;Schäfer and Turek 1996)

Full Order Model Approximation
After obtaining the trained neural network, we use the ROM dynamic system (16) to approximate the full model of NSE (1). The following average L 2 error formula has been used to quantify the accuracy of the model, We first evaluate the model with different layers and neurons. The dimension of the ROM dynamics is fixed to be r = 8 with the one step Adams Moulton method. Table 1 provides a crude estimate that increasing network width (256 neurons) might have the potential over-fitting issue whereas decreasing units (64) may not be enough to reach a good accuracy. Also, the network depth (number of hidden layers) have a positive effect on the performance of the model. To reduce the numerical efforts, we use one hidden layer and 128 neurons for the neural network in the rest of our evaluations. We emphasize that to fully understand the model sensitivity with respect to network architecture, a systematic study involving regularization, batch normalization and drop out is needed in future research. We also test the LMNet-ROM with different steps in the training. Table 2 lists the error between the new model and the exact data. As a comparison, we add GP-ROM result in the last entry. Table 2 shows that the LMNet-ROM consistently provide more accurate results as increasing steps of Adams Moulton (AM) method. An intuitive explanation is that the stability property of AM method helps to regularize the network and eventually achieve a good calibration. Large steps (K) requires high computational cost of training the network. To balance the output accuracy and training cost, we use step K = 1 for the rest of our numerical tests. 3.79e-04 6.20e-04 6.37e-04 GP-ROM 1.66e-01 7.30e-02 1.92e-02 Noisy data. The above numerical tests are carried out on the deterministic data. In some situations, however, problems may contain noise measurements. We study the robustness of the new method with respect to noise data by adding Gaussian noise to the data-set for both models. Table 3 lists the error comparison between the LMNet-ROM and GP-ROM for different level of noise data. The results show that the LMNet-ROM cannot preserve good performance when the noise magnitude (≥1%) is high while GP-ROM does. The argument is that the learned dynamical system fully depends on the data which makes it vulnerable to noise interference. The GP-ROM, however, requires the use of FOM operators making it less sensitive than LMNet-ROM to noise. Further approaches should be investigated for the potential improvement for this problem. As for deterministic data, the LMNet-ROM is better than GP-ROM in full system approximation.

Long-Term Prediction
In this section, we make a thorough study of long-term predictability of the new LMNet-ROM. The solution to our new model and the GP-ROM are computed by the dynamic system (16) and (6), respectively. We use direct numerical simulation (DNS) to denote the exact solution (data) of NSE. We use snapshots data that are collected from the time interval T = [0, 3] to generate the POD space and train the neural network. We then run the reduced systems (16) and (6) for T = [0, 5] to make the prediction. Fig. 3 plots the phase portraits of the first few coefficients, a 2 , a 3 , a 4 , from both models and the DNS data. The red line that depicts the result from LMNet-ROM has a closer mimic of DNS data, whereas the portraits from the GP-ROM have small deviation. This behavior tells that the dynamics of LMNet-ROM predict future states better than GP-ROM with the given information. Note that a 1 is a constant and not meaningful to discuss since the first POD mode ϕ 1 represents the mean flow. We also looked at the prediction of the time evolution of energy (E(t j ) = 1 2 u j L 2 ), vorticity construction, and drag coefficients. Fig. 4 shows the long-term prediction of energy travel and drag. The main observation is that the LMNet-ROM performs much better than GP-ROM. The energy and drag generated from GP-ROM have a huge deterioration when time evolves, which means the prediction is not accurate. Clearly, the prediction from LMNet-ROM is impressively good as it is stable and close to DNS, see Fig. 4. Vorticity describes the local spinning motion of a fluid system. Fig. 5 plots the vorticity construction from the velocity field around the cylinder at the end time T = 5. As depicted in Fig. 5, the LMNet-ROM correctly predicts the vortex street behind the cylinder while GP-ROM not. The above results are presented for dimension r = 8, but similar results can be found for r = 4, 6. Overall, the long-term predictability of LMNet-ROM is much better than GP-ROM.

LMNet-ROM vs. Closure Models
In this section, we present a preliminary numerical comparison between the LMNet-ROM and the closure models. Due to the wide class of stabilization closure models and the availability of their open source implementation, a thorough comparison between our new model and all other models is not practical. Therefore, we use the two most recent models (open source implemenation), Data-Driven-Filtered ROM (DDF-ROM) and Evolve-then-Filter ROM (EF-ROM), to illustrate the numerical comparison. The EF-ROM uses a two steps regularization strategy to improve the GP-ROM (Wells . The DDF-ROM solves an optimization problem to approximate the τ in system (8) (Xie et al. 2018). Table 4 lists the average L 2 error from the three models with different dimensions. It surprisingly indicates that the closure models with mathematical methods cannot outperform the LMNet-ROM. This result is impressive, given that the LMNet-ROM is learned through the neural network from the data without acquiring any FOM information. We note that the goal of both the stabilization closure models and our new model is to provide an accurate reduced dynamics to approximate the full system from different perspective. The former generally model the artificial term (τ ) in the system (8) physically or mathematically, whereas the latter use data to learn the dynamics (16). We claim that the non-intrusive learning framework has the generality that can be applied to any nonlinear fluid system since it only requires the training data.

Computational Cost
In this section, we discuss the computational efficiency of the proposed LMNet-ROM. The main computational cost of reduced order models is the offline computation since the cost of solving a small ODE system is negligible at the online stage. The FOM simulation (DNS) time is used as a benchmark to evaluate the performance of each model. The  Table 5 lists the CPU time from each ROM and the associated speed-up factor. The LMNet-ROM time is only computed from one step AM method with one layer and 128 neurons, given the fact that this network architecture achieves good accuracy in the previous test. The result in Table 5reveals that the LMNet-ROM is more efficient compared to other models. This is a huge advantage of our new method since reducing the computational cost while maintaining good accuracy is the primary goal of reduced order modeling.

Conclusions and Outlook
In this paper, we proposed a novel learning reduced order model framework for the numerical simulation of fluid flows. This framework was based on the recent development of linear multistep network architecture. We numerically studied the LMNet-ROM in the simulation of a 2D flow past a cylinder. The numerical results demonstrate that the LMNet-ROM was significantly more accurate than the GP-ROM in system approximation and long-term prediction. Furthermore, we compared the new model with the two most recent stabilization closure models, EF-ROM and DDF-ROM. The results show that our new method outperforms the two closure models. Overall, the LMNet-ROM beats the aforementioned models both in accuracy and computational efficiency, which provides a promising and encouraging approach in model reduction of fluid dynamics. However, the LMNet-ROM's potential still needs to be explored. We outline some research directions that could be pursued.
Probably the most important next step is to study the parametrized system prediction of the LMNet-ROM. The current neural network is trained under the data-set given by one parameter value (Re) from the full order model of NSE. How does the LMNet-ROM predict systems with different parameter values, such as initial conditions and boundary conditions? Parametrized system prediction is a challenging problem in engineering applications. We hope to provide a systematic investigation with the new LMNet-ROM in later research.
Another important research direction is to improve the robustness of the model with respect to noise data. Table  3 shows the drawback of this model for high magnitude noisy (≥1%) data. We plan to address this issue by improving the neural network architecture. Also, regularization for preventing over-fitting needs to be fully studied in future research.
Finally, the generality study of LMNet-ROM is worth investigation. Although we constructed and tested the LMNet-ROM in a fluid dynamics setting, the LMNet-ROM framework can be applied to any type of nonlinear partial differential equations (PDE) that is amenable to reduced order modeling. The only input needed in the LMNet-ROM framework is the data from the FOM of any system, see Algorithm 1. The LMNet-ROM procedure does not restrict it to the particular physical system modeled by the given nonlinear PDE. Since the LMNet-ROM is built by fully data-driven learning, we expect it to be successful in the numerical simulation of general mathematical models (e.g., from elasticity or bioengineering).