MRT-Lattice Boltzmann Model for Multilayer Shallow Water Flow

Abstract

The objectives of this study are to introduce a multiple-relaxation-time (MRT) lattice Boltzmann model (LBM) to simulate multilayer shallow water flows and to introduce graphics processing unit (GPU) computing to accelerate the lattice Boltzmann model. Using multiple relaxation times in the lattice Boltzmann model has an advantage of handling very low kinematic viscosity without causing a stability problem in the shallow water equations. This study develops a multilayer MRT-LBM to solve the multilayer Saint-Venant equations to obtain horizontal flow velocities in various depths. In the multilayer MRT-LBM, vertical kinematic viscosity forcing is the key term to couple adjacent layers. We implemented the multilayer MRT-LBM to a GPU-based high-performance computing (HPC) architecture. The multilayer MRT-LBM was verified by analytical solutions for cases of wind-driven, density-driven, and combined circulations with non-uniform bathymetry. The results show good speedup and scalability for large problems. Numerical solutions compared well to the analytical solutions. The multilayer MRT-LBM is promising for simulating lateral and vertical distributions of the horizontal velocities in shallow water flow.

1. Introduction

The shallow water equations are used to describe flow in bodies of water where the horizontal length scales are much greater than the fluid depth (e.g., river or lake hydrodynamics, coastal and estuarine circulation, overland flow, etc.). They have wide applications in hydraulic engineering [1,2,3] and coastal engineering [4], and can be used to study main physical phenomena of interest to scientists and engineers such as storm surges [5], tsunami and bore wave propagation [6], the stationary hydraulic jump, forces acting on off-shore structures, and river, reservoir and open channel flows [1,7]. The shallow water equations can also be coupled to transport equations to model pollutant transport [8], salinity and temperature [9], and sediment transport [6,10], which are important subjects in many industrial and environmental projects. More advanced depth-averaged models were developed for shear shallow water flows [11,12] and for coastal waves in the shoaling and surf zones [13,14].

When vertical effects are important, such as in baroclinic regimes, where density varies with salinity and temperature, three-dimensional flow should be used. This would require the solution of a system of equations coupling the Navier–Stokes equation to a moving free surface boundary. However, solving full Navier–Stokes equations in three dimensions is computationally challenging and may have difficulties handling the discontinuities in the free surface. Substantial literature exists on the application of various numerical methods, e.g., finite difference methods (FDMs), finite volume methods (FVMs), and finite element methods (FEMs) to the three-dimensional shallow water equations [15,16,17].

The multilayer shallow water equations under the hydrostatic assumption present an alternative solution to the free surface Navier–Stokes system and lead to a precise description of the vertical profile of the horizontal velocity while preserving the robustness and computational efficiency of the shallow water equations. This study introduces an emerging mesoscopic numerical method, the lattice Boltzmann model, to simulate multilayer shallow water flows.

1.1. Lattice Boltzmann Model

Modeling of problems in hydrodynamics, hydraulics, and environmental fluid mechanics may be undertaken at three different length scales, commonly referred to as the microscopic, mesoscopic, and macroscopic levels [18]. Mesoscopic models based on the Boltzmann equation can be categorized into two sub-classes: continuous Boltzmann models and discrete Boltzmann models such as the lattice Boltzmann model. The main difference is that the distribution function is either continuous or discrete in particle velocity.

The lattice Boltzmann model can be seen as an alternative numerical scheme for simulating a wide range of fluid dynamics. The lattice Boltzmann model (LBM) was originally created to model flows governed by the Navier–Stokes equations [19,20,21,22,23]. This is because LBM is simpler to formulate, apply, and implement, including in high-performance computing (HPC) environments, than Riemann solvers because they do not require characteristic decomposition.

More recently, the Boltzmann-based methods have developed into an alternative and promising numerical technique for a wide range of computational fluid dynamics (CFD) techniques [24], including shock waves in compressible flows [25], multicomponent and multiphase flows in complex geometries [19,24], transcritical flows [26] and turbulent flows [27]. The LBM to porous medium flow includes Richard’s unsaturated flow equation [28,29] and flow and mass transport [30,31,32,33]. The LBM has found a wide range of applications in a variety of fields involving shallow water equations. The LBM has been successfully adopted to study shallow water equations of wind-driven ocean circulation [34], intrusion of salt wedges in estuarine zones [35], continuous change in bed elevation [36], wetting–drying problems [37], two immiscible shallow layers of different density [38], and multilayer shallow water equations [39].

1.2. LBM on HPC Environments

The advantages of the LBM are its ease in parallelization because of the locality of particle interaction and the transport of particle information. The implementation of LBM on central processing unit (CPU)-based systems has been researched and numerous improvements are possible starting from standard LBM implementations [40,41]. The implementation of LBM on CPU-based architectures is achieved on both distributed and shared memory systems. LUDWIG [42] is a parallel LBM code for fluids, implementing message passing interface (MPI) to achieve full portability and good efficiency on both massively parallel processors (MPP) and symmetric multiprocessing (SMP) systems. With OpenMP [43], the LBM has been optimized to be implemented on multiple CPUs with shared-memory parallel programming [44]. More recently, the LBM has been a good candidate for implementation on hardware-accelerated systems using Graphics Processing Units (GPUs). It has been accelerated on a single GPU [45,46] or a GPU cluster [47,48] with MPI using the Compute Unified Device Architecture (CUDA).

1.3. Objective of the Study

The objective of this study is to develop a multiple-relaxation time (MRT)-LBM to simulate multilayer shallow water equations under GPU high-performance computing. This attempt is essential to extend the full capability of LBM to shallow water flow studies. Tubbs and Tsai [46] showed that single-relaxation-time (SRT)-LBM has a serious stability problem for solving shallow water equations with very small viscosity. This stability would be even worse in the multilayer shallow water equations. This study will introduce MTR-LBM to increase stability and accuracy and eliminate spurious oscillations. Moreover, this study aims to investigate LBM performance on GPU-based HPC environments using MATLAB code and the Jacket GPU engine.

2. Multilayer Shallow Water Equations

Consider a shallow water flow regime in which the vertical length scale is much smaller than the horizontal length scale. By depth-integrating the continuity equation and the Navier–Stokes equations for incompressible and viscous flows with free surface, the shallow water equations with horizontal viscous terms are [3]:

$$\frac{\partial h}{\partial t}+\frac{\partial \left(h{u}_i\right)}{\partial x_i}=0$$

(1)

$$\frac{\partial \left(h{u}_i\right)}{\partial t}+\frac{\partial \left(h{u}_i{u}_j\right)}{\partial x_j}+\frac{\partial }{\partial x_i}\left(\frac{1}{2}g{h}^2\right)=\nu \left[\frac{{\partial }^2\left(h{u}_i\right)}{\partial x_j\partial x_j}\right]+F_i$$

(2)

where i and j are Cartesian indices and the Einstein summation convention is used, h is the water depth, u_i is the depth-averaged velocity component in the i direction, g is the gravitational acceleration, $\nu$ is the kinematic viscosity, F_i is the external force acting on the shallow water flow, and t is the time. Based on the multilayer Saint-Venant system [49,50], the governing equations are similar to the above shallow water equations with horizontal viscous terms for each layer and additional terms for transferring momentum between layers:

$$\frac{\partial h^{(\ell )}}{\partial t}+\frac{\partial \left(h^{(\ell )}{u}_i^{(\ell )}\right)}{\partial x_i}=0$$

(3)

$$\frac{\partial \left(h^{(\ell )}{u}_i^{(\ell )}\right)}{\partial t}+\frac{\partial \left(h^{(\ell )}{u}_i^{(\ell )}{u}_j^{(\ell )}\right)}{\partial x_j}+\frac{\partial }{\partial x_i}\left(\frac{1}{2}g{h}^{(\ell )}H\right)=\nu \left[\frac{{\partial }^2\left(h^{(\ell )}{u}_i^{(\ell )}\right)}{\partial x_j\partial x_j}\right]+F_i^{(\ell )},\ell =1,2,\dots ,M$$

(4)

where $h_{}^{(\ell )}$ is the local water height in layer $\ell$, $u_i^{(\ell )}$ is the local velocity component in the i direction in layer $\ell$, $F_i^{(\ell )}$ is the external force acting on layer $\ell$ $H={\displaystyle {\sum }_{m=1}^{M_L}{h}^{(m)}}$ is the water depth, and M_L is the number of layers.

Six external forces are included in the model. They are wind-driven forcing ($F_{Wi}^{(\ell )}$) at the top-most layer, bed slope forcing ($F_{Pi}^{(\ell )}$), vertical kinematic eddy viscosity forcing ($F_{\mu i}^{(\ell )}$), non-conservative pressure source ($F_{NCi}^{(\ell )}$) [49,50,51], density gradient forcing, $\left(F_{\rho }^{\left(\ell \right)}\right)$ [52], and forcing from the Coriolis effect ($F_{Ci}^{(\ell )}$) as follows:

$$F_i^{(\ell )}=F_{Wi}^{(\ell )}+F_{Pi}^{(\ell )}+F_{\mu i}^{(\ell )}+F_{NCi}^{(\ell )}+F_{\rho i}^{\left(\ell \right)}+F_{Ci}^{(\ell )}$$

(5)

$$F_{Wi}^{(\ell )}={\delta }_{M\ell }\frac{{\tau }_i^W}{\rho }={\delta }_{M\ell }\frac{{\rho }_a{C}_W{U}_{Wi}{W}_s}{\rho }$$

(6)

where ${\delta }_{M\ell }$ is the Kronecker delta function (${\delta }_{M\ell }$ = 1, if $\ell =M$) and ${\tau }_i^W$ is the wind stress in i direction, which is the product of fluid density (ρ), air density (ρ_a), wind stress coefficient (C_W), wind speed measured at 10 m above water surface (W_s), and wind velocity component in i direction (U_Wi).

$$F_{Pi}^{(\ell )}=-g{h}^{(\ell )}\frac{\partial z_b}{\partial x_i}$$

(7)

where $z_b$ is the bed elevation.

$$F_{\mu i}^{(\ell )}=-\kappa {\delta }_{1\ell }{u}_i^{(\ell )}+2\mu \left(1-{\delta }_{M\ell }\right)\frac{u_i^{(\ell +1)}-u_i^{(\ell )}}{h_{}^{(\ell +1)}+h_{}^{(\ell )}}-2\mu \left(1-{\delta }_{1\ell }\right)\frac{u_i^{(\ell )}-u_i^{(\ell -1)}}{h_{}^{(\ell )}+h_{}^{(\ell -1)}}$$

(8)

where ${\delta }_{1\ell }$ is Kronecker delta function (${\delta }_{1\ell }$ = 1, if $\ell =1$), $\kappa$ is the bottom friction coefficient, and μ is the vertical (kinematic) eddy viscosity, which is also called vertical viscosity [49,50,51] or internal friction coefficient [52]. The linear bed friction law is chosen to calculate bed friction. The 2nd and 3rd terms on the right-hand side of Equation (8) represent internal friction between layers. The vertical eddy viscosity μ is much larger than the kinematic viscosity $\nu$ in Equation (4).

$$F_{NCi}^{(\ell )}=\frac{g{H}^2}{2}\frac{\partial }{\partial x_i}\left(\frac{h^{(\ell )}}{H}\right)$$

(9)

$$F_{\rho i}^{\left(\ell \right)}=\frac{g{h}^{\left(\ell \right)}}{\rho }{\displaystyle {\int }_H^{\overline{z}}\frac{\partial \rho }{\partial x_i}dz}$$

(10)

where $\overline{z}=\frac{1}{2}{h}^{\left(\ell \right)}+{\displaystyle {\sum }_{m=1}^{\ell -1}{h}^{\left(m\right)}}$ is at the center of layer $\ell$. The fluid density is constant in this study. However, this study follows [52] to include longitudinal density gradient in order to study the baroclinic circulation.

$$F_{Ci}^{(\ell )}=\{\begin{array}{cc}{f}_c{h}^{\left(\ell \right)}{u}_y,& i=x\hfill \\ -f_c{h}^{\left(\ell \right)}{u}_x,& i=y\hfill \end{array}$$

(11)

where $f_c=2\varpi \sin \phi$ is the Coriolis parameter, which a function of Earth rotation rate ($\varpi$) and latitude ($\phi$).

3. MRT-Lattice Boltzmann Modeling

This study adopts the multiple-relaxation-time (MRT) lattice Boltzmann model [53,54] to solve the multilayer shallow water equations. Specifically, the authors implement the MRT-LBM to a D2Q9 lattice, which defines the streaming velocity as

$$c_{\alpha }=\{\begin{array}{ll}\left(0,0\right)\hfill & \alpha =0\hfill \\ c\left[cos\left(\frac{1}{4}\left(2\alpha -2\right)\pi \right),sin\left(\frac{1}{4}\left(2\alpha -2\right)\pi \right)\right]& \alpha =1,2,3,4\hfill \\ \sqrt{2}c\left[cos\left(\frac{1}{4}\left(2\alpha -9\right)\pi \right),sin\left(\frac{1}{4}\left(2\alpha -9\right)\pi \right)\right]& \alpha =5,6,7,8\hfill \end{array}$$

(12)

where $c_{\alpha }=\left\{c_{\alpha i}\right\}$ is a streaming velocity along a streaming direction defined by an index α, and c is the lattice speed. Given square lattice size ∆x and time step ∆t, the lattice speed is c = ∆x/∆t. Then, the evolution equation for the MRT-LBM on a D2Q9 lattice for each layer $\ell$ is

$$f^{(\ell )}\left(x_i+c_{\alpha i}\Delta t,t+\Delta t\right)-f^{(\ell )}\left(x_i,t\right)=-M^{-1}S\left[m^{(\ell )}\left(x_i,t\right)-m^{(\ell )}{}^{eq}\left(x_i,t\right)\right]+\frac{\Delta t}{6c^2}{F}_{\alpha }^{(\ell )}\left(x_i,t\right)$$

(13)

where $f^{(\ell )}=\{f_{\alpha }^{(\ell )},\alpha =0,1,2,\dots ,8\}\in R^9$ is a vector of particle distribution functions for layer $\ell$ $m^{(\ell )}\in R^9$ and $m^{(\ell )}{}^{eq}\in R^9$ are vectors of moments and their equilibria, respectively, for layer $\ell$, and $M\in R^{9\times 9}$ is a transformation matrix that transforms the particle distribution functions and equilibrium distribution functions (EDFs) from velocity space to moment space, which makes $m^{(\ell )}=M{f}^{(\ell )}$ and $m^{(\ell )}{}^{eq}=M{f}^{(\ell )}{}^{eq}$. $f^{(\ell )}{}^{eq}=\{f_{\alpha }^{(\ell )eq},\alpha =0,1,2,\dots ,8\}\in R^9$ is a vector of the EDFs for layer $\ell$. $S=diag\left(s_0,s_1,\dots ,s_8\right)\in R^{9\times 9}$ is a diagonal matrix of multiple relaxation rates, where $s_{\alpha }$ are the relaxation rates. $F_{\alpha }^{(\ell )}=\{{\displaystyle {\sum }_i{c}_{\alpha i}{F}_i^{(\ell )}},\alpha =0,1,2\dots ,8\}$ is a vector of external forces along the α direction. The forcing terms $F_i^{(\ell )}$ are given by the central scheme [3].

The evolution Equation (13) consists of two steps: streaming and collision. At the left-hand side, particle transport is achieved by pure advection executed in the streaming velocity space. At the right-hand side, particle collision is achieved by linear relaxation processes executed in the moment space. For each time step, particle distribution functions reach their neighboring nodes simultaneously through prescribed lattice connections. The transformation matrix M for D2Q9 is given by Lallemand and Luo [22]:

$$M=\left(\begin{array}{r}\hfill b_0^T\\ \hfill b_1^T\\ \hfill b_2^T\\ \hfill b_3^T\\ \hfill b_4^T\\ \hfill b_5^T\\ \hfill b_6^T\\ \hfill b_7^T\\ \hfill b_8^T\end{array}\right)=\left(\begin{array}{rrrrrrrrr}\hfill 1& \hfill 1& \hfill 1& \hfill 1& \hfill 1& \hfill 1& \hfill 1& \hfill 1& \hfill 1\\ \hfill -4& \hfill -1& \hfill -1& \hfill -1& \hfill -1& \hfill 2& \hfill 2& \hfill 2& \hfill 2\\ \hfill 4& \hfill -2& \hfill -2& \hfill -2& \hfill -2& \hfill 1& \hfill 1& \hfill 1& \hfill 1\\ \hfill 0& \hfill 1& \hfill 0& \hfill -1& \hfill 0& \hfill 1& \hfill -1& \hfill -1& \hfill 1\\ \hfill 0& \hfill -2& \hfill 0& \hfill 2& \hfill 0& \hfill 1& \hfill -1& \hfill -1& \hfill 1\\ \hfill 0& \hfill 0& \hfill 1& \hfill 0& \hfill -1& \hfill 1& \hfill 1& \hfill -1& \hfill -1\\ \hfill 0& \hfill 0& \hfill -2& \hfill 0& \hfill 2& \hfill 1& \hfill 1& \hfill -1& \hfill -1\\ \hfill 0& \hfill 1& \hfill -1& \hfill 1& \hfill -1& \hfill 0& \hfill 0& \hfill 0& \hfill 0\\ \hfill 0& \hfill 0& \hfill 0& \hfill 0& \hfill 0& \hfill 1& \hfill -1& \hfill 1& \hfill -1\end{array}\right)$$

(14)

where the vectors $\left\{b_{\alpha }\right\}$ are mutually orthogonal [53,55]. Inserting matrices M and S into Equation (13), the evolution equation in the direction α for each layer becomes [54]

$$f_{\alpha }^{(\ell )}\left(x_i{+c}_{\alpha i}\Delta t,t+\Delta t\right){=f}_{\alpha }^{(\ell )}\left(x_i,t\right)-{\displaystyle \sum _{\beta =0}^8\frac{s_{\beta }{b}_{\beta \alpha }}{\Vert b_{\beta }\Vert }\left[m_{\beta }^{(\ell )}\left(x_i,t\right)-m_{\beta }^{(\ell )eq}\left(x_i,t\right)\right]}+\frac{\Delta t}{6c^2}{\displaystyle \sum _i{c}_{\alpha i}{F}_i^{(\ell )}\left(x_i,t\right)}$$

(15)

The moments $m^{(\ell )}$ applied to the shallow water equations for each layer are

$$m^{(\ell )}=\left(h^{(\ell )}{,e}^{(\ell )}{,\epsilon }^{(\ell )}{,j}_x^{(\ell )}{,q}_x^{(\ell )}{,j}_y^{(\ell )}{,q}_y^{(\ell )}{,p}_{xx}^{(\ell )}{,p}_{xy}^{(\ell )}\right)$$

(16)

where $m_0^{(\ell )}=h^{(\ell )}$ is the water depth, $m_1^{(\ell )}=e^{(\ell )}$ is related to the total energy, $m_2^{(\ell )}={\epsilon }^{(\ell )}$ is related to the energy square, $\left(m_3^{(\ell )},m_5^{(\ell )}\right)$ = $\left(j_x^{(\ell )},j_y^{(\ell )}\right)$ = $\left(h{u}_x^{(\ell )},h{u}_y^{(\ell )}\right)$ are the flow momenta, $\left(m_4^{(\ell )},m_6^{(\ell )}\right)$ = $\left(q_x^{(\ell )},q_y^{(\ell )}\right)$ are related to the head flux, and $m_7^{(\ell )}=p_{xx}^{(\ell )}$ $m_8^{(\ell )}=p_{xy}^{(\ell )}$ are related to the diagonal and off-diagonal components of the stress tensor, respectively. When applied to the shallow water equations, the conserved moments are the water depth and the flow momenta in each layer:

$$m_0^{(\ell )}=h^{(\ell )}={\displaystyle {\sum }_{\alpha =0}^8{f}_{\alpha }^{(\ell )}}$$

(17)

$$m_3^{(\ell )}=h^{(\ell )}{u}_x^{(\ell )}={\displaystyle {\sum }_{\alpha =0}^8{c}_{\alpha x}{f}_{\alpha }^{(\ell )}},m_5^{(\ell )}=h^{(\ell )}{u}_y^{(\ell )}={\displaystyle {\sum }_{\alpha =0}^8{c}_{\alpha y}{f}_{\alpha }^{(\ell )}}.$$

(18)

The remaining moments are not conserved quantities. The EDFs applied to the multilayer shallow water are Equation (17).

$$\begin{array}{l}{f}_{\alpha }^{(\ell )eq}{=h}^{(\ell )}{\omega }_{\alpha }\left(\frac{c_s^2}{c^2}+\frac{u_i^{(\ell )}{c}_{\alpha i}}{c^2}+\frac{3}{2}\frac{{\left(u_i^{(\ell )}{c}_{\alpha i}\right)}^2}{c^4}-\frac{1}{2}\frac{u_i^{(\ell )}{u}_i^{(\ell )}}{c^2}\right),\alpha >0\\ f_0^{(\ell )eq}{=h}^{(\ell )}-{\displaystyle \sum _{\alpha =1}^8{f}_{\alpha }^{(\ell )eq}}\end{array}$$

(19)

where $c_s^2=gH/2$ is known as the squared speed of sound and is obtained through the recovery of the shallow water equations up to second order by the Chapman–Enskog analysis. For the D2Q9 lattice, the weighting coefficient ${\omega }_{\alpha }$ = 1/3 is for α = 1,2,3,4 and ${\omega }_{\alpha }$ = 1/12 is for α = 5,6,7,8. Since each layer has the same planar discretization, the weighting coefficients ${\omega }_{\alpha }$ remain the same for each layer. Using Equations (14) and (19), the equilibrium moments, $m^{(\ell )}{}^{eq}=M{f}^{(\ell )}{}^{eq}$, are [46]

$$m_1^{(\ell )eq}=-4h^{(\ell )}+\frac{3g{h}^{(\ell )2}}{c^2}+\frac{3h^{(\ell )}\left(u_x^{(\ell )2}+u_y^{(\ell )2}\right)}{c^2},m_2^{(\ell )eq}=4h^{(\ell )}-\frac{9g{h}^{(\ell )2}}{2c^2}-\frac{3h^{(\ell )}\left(u_x^{(\ell )2}+u_y^{(\ell )2}\right)}{c^2}$$

(20)

$$m_4^{(\ell )eq}=-\frac{h^{(\ell )}{u}_x^{(\ell )}}{c},m_6^{(\ell )eq}=-\frac{h^{(\ell )}{u}_y^{(\ell )}}{c}$$

(21)

$$m_7^{(\ell )eq}=\frac{3h^{(\ell )}\left(u_x^{(\ell )2}-u_y^{(\ell )2}\right)}{c^2},m_8^{(\ell )eq}=\frac{h^{(\ell )}{u}_x^{(\ell )}{u}_y^{(\ell )}}{c^2}$$

(22)

The equilibria of the conserved moments ($m_0^{(\ell )}$, $m_3^{(\ell )}$, and $m_5^{(\ell )}$) are equal to themselves. Therefore, the relaxation rates s₀, s₃, and s₅ have no effect on MRT-LBM solutions. With the moment equilibria given by Equations (20)–(22), the shallow water equations can be recovered with the shear viscosity ($\nu$) and bulk viscosity ($\zeta$) given [55] as

$$v=\frac{1}{3}\left(\frac{1}{s_7}-\frac{1}{2}\right)c\Delta x=\frac{1}{3}\left(\frac{1}{s_8}-\frac{1}{2}\right)c\Delta x$$

(23)

$$\zeta =\frac{1}{6}\left(\frac{1}{s_1}-\frac{1}{2}\right)c\Delta x$$

(24)

Setting $s_{\alpha }=1/\tau$, the evolution Equation (13) reduces to the SRT-LBM model, where τ is the relaxation time. The LBM with a single relaxation time is commonly referred to as the Bhatnagar–Gross–Krook (BGK) collision operator model (LBGK), and has been introduced to the shallow water equations [34,56]. For the SRT-LBM, it has $\nu =2\zeta$$=\frac{1}{3}\left(\tau -\frac{1}{2}\right)c\Delta x$. When τ is close to 1/2, the kinematic viscosity becomes very small (e.g., $\nu =1\times {10}^{-6}$ m²/s), causing the numerical instability. The MRT-LBM model has no such problem because the relaxation rates can be selected to attain stable solutions. In addition, to increase solution stability, this study adopts an implicit step to update flow velocities [39].

4. Multilayer Initial and Boundary Conditions

4.1. Initial Conditions

The initial conditions for a physical problem to be modeled are given in the form of macroscopic variables, which is normal practice in traditional numerical methods. Since the lattice Boltzmann formulation is based on solving Equation (15), the initial conditions must be written in terms of the distribution function. Given the initial macroscopic boundary conditions, $h^{(\ell )}\left(x_i,t=0\right)$, $u_x^{(\ell )}\left(x_i,t=0\right)$, and $u_y^{(\ell )}\left(x_i,t=0\right)$, the EDFs, $f_{\alpha }^{(\ell )eq}$ (Equation (19)), are computed and used as initial conditions for $f_{\alpha }^{(\ell )}$.

4.2. Periodic Boundary Conditions

In cases where the flow pattern repeats itself at boundaries, a periodic boundary condition is required. To achieve boundary conditions in the x direction in the lattice Boltzmann formulation, we set the unknown distribution functions, $f_1^{(\ell )}$, $f_5^{(\ell )}$ and $f_8^{(\ell )}$ at the inflow boundary, ${\Gamma }_{inflow}$ (see Figure 1a for an example) to be the same as known distribution functions at the outflow boundary, ${\Gamma }_{outflow}$:

$$f_{\alpha }^{(\ell )}\left(x_i\in {\Gamma }_{inflow},t\right){=f}_{\alpha }^{(\ell )}\left(x_i\in {\Gamma }_{outflow},t\right),\alpha =1,5,8$$

(25)

The unknown distribution functions, $f_3$, $f_6$ and $f_7$ at the outflow boundary are set to the corresponding known distribution functions at the inflow boundary:

$$f_{\alpha }^{(\ell )}\left(x_i\in {\Gamma }_{outflow},t\right){=f}_{\alpha }^{(\ell )}\left(x_i\in {\Gamma }_{inflow},t\right),\alpha =3,6,7$$

(26)

Periodic boundary conditions similar to Equations (25) and (26) in the y direction can be formulated.

4.3. Solid Boundary Conditions

Solid boundaries in the flow region can be either no-slip or free-slip, which corresponds to zero velocity or zero normal velocity, respectively, at the boundary. A no-slip boundary condition is achieved by the bounce-back scheme under symmetry conditions [21]. We set the unknown distribution functions, $f_2^{(\ell )}$, $f_5^{(\ell )}$ and $f_6^{(\ell )}$ at the solid boundary (see Figure 1b for example) to the known distributions, $f_4^{(\ell )}$, $f_7^{(\ell )}$ and $f_8^{(\ell )}$ corresponding to the opposite directions:

$$f_2^{(\ell )}=f_4^{(\ell )},f_5^{(\ell )}=f_7^{(\ell )},f_6^{(\ell )}=f_8^{(\ell )}.$$

(27)

This results in zero normal and tangential flow velocities at the boundary.

A free-slip boundary condition is also achieved by the bounce-back scheme [21]. The unknown distribution functions, $f_2^{(\ell )}$, $f_5^{(\ell )}$ and $f_6^{(\ell )}$ at the solid boundary is assigned to the known distributions, $f_4^{(\ell )}$, $f_8^{(\ell )}$, and $f_7^{(\ell )}$ corresponding to the reflected directions to the solid boundary:

$$f_2^{(\ell )}=f_4^{(\ell )},f_5^{(\ell )}=f_8^{(\ell )},f_6^{(\ell )}=f_7^{(\ell )}.$$

(28)

This ensures no flow across the normal direction and flow along the tangential direction.

4.4. Open Boundary Conditions

Open boundary conditions refer to known macroscopic boundary values or functions at boundaries (e.g., known water depth at inflow or outflow boundaries). If flow velocity and water depth at the boundaries are known, unknown distribution functions can be computed using Equations (17) and (18) following the method described by Zou and He [57]. For example, at the inflow boundary (see Figure 1a), Equations (17) and (18) lead to the following three equations:

$$f_0^{(\ell )}+f_1^{(\ell )}+f_2^{(\ell )}+f_3^{(\ell )}+f_4^{(\ell )}+f_5^{(\ell )}+f_6^{(\ell )}+f_7^{(\ell )}+f_8^{(\ell )}=h^{(\ell )}$$

(29)

$$c\left(f_1^{(\ell )}+f_5^{(\ell )}+f_8^{(\ell )}\right)-c\left(f_3^{(\ell )}+f_6^{(\ell )}+f_7^{(\ell )}\right)=h^{(\ell )}{u}_x^{(\ell )}$$

(30)

$$c\left(f_2^{(\ell )}+f_5^{(\ell )}+f_6^{(\ell )}\right)-c\left(f_4^{(\ell )}+f_7^{(\ell )}+f_8^{(\ell )}\right)=h^{(\ell )}{u}_y^{(\ell )}$$

(31)

Given known flow velocities and water heights in each layer in Equations (29)–(31), three unknown distribution functions, $f_1^{\left(\ell \right)}$, $f_5^{\left(\ell \right)}$ and $f_8^{\left(\ell \right)}$ can be determined. The same approach is applicable to determine three unknown distribution functions, $f_3^{\left(\ell \right)}$, $f_6^{\left(\ell \right)}$ and $f_7^{\left(\ell \right)}$, at the outflow boundary.

5. GPU Accelerated LBM

This study implements the MRT-LBM code to a GPU architecture to solve the multilayer shallow water equations. The MRT-LBM code was written in MATLAB^® (2008, Natick, MA, USA) and is parallelized with AccelerEyes Jacket GPU Engine and the NVIDIA^® Compute Unified Device Architecture (CUDA) [58] on a single GPU workstation. In all simulations, the computations were performed in a single precision. In this study, the simulations are performed on a single workstation 3.0-GHz Intel^® Core™2 Extreme quad core with an NVIDIA^® Tesla™ C1060 Computing Processor. The NVIDIA^® Tesla™ C1060 Computing Processor contains 240 stream processors running at 1.3 GHz, which has a peak performance of 933 GFLOPs. Readers refer to [46] for detail explanations of how the GPU accesses memory and processes data.

5.1. Jacket’s GPU Engine

The Jacket GPU engine for MATLAB^® is built on NVIDIA’s CUDA technology. It enables a standard MATLAB code to run on the GPU and connects the user-friendliness of MATLAB directly to the speed and visual computing capability of the GPU [59]. MATLAB GPU computing with Jacket starts at the most basic level through the replacement of low-level MATLAB data structures which normally reside on the CPU with data structures that reside on the GPU. This replaces the lowest level of MATLAB’s CPU-bound computation engine, moving the work MATLAB would normally do on the CPU to the GPU. Jacket Beta version 0.3-20080710 [59] on a 32-bit Windows XP with MATLAB R2007b is used. Jacket is run on the 2.0 beta version of the CUDA toolkit for Windows, which uses version 1.1 compute capabilities.

Jacket-enabled MATLAB scripts achieve speed improvements in the range of 2–10 times, and in some cases up to 100x improvements, over equivalent CPU versions. While Jacket accelerates MATLAB functions and computations at a lower level, the overall speedup of an algorithm depends on the nature of the algorithm. The LBM has a very straightforward implementation consisting of only local calculations (collisions) and nearest neighbor memory transfers (streaming), which makes it a great candidate to be implemented both on the GPU and in MATLAB.

5.2. Optimizing MATLAB GPU Performance

Implementing algorithms on the GPU using Jacket requires certain considerations to optimize performance. Both MATLAB and Jacket perform best on vectorized code. A vectorized code can make it easy to determine which parts of an algorithm are inherently serial or parallel. Both MATLAB and Jacket take advantage of the inherent parallelism of the MATLAB scripting M-language which is extremely powerful when utilized wisely. A good understanding of the memory dependencies of an algorithm is necessary as CPUs are inherently serial computing devices and GPUs are inherently parallel computing devices. In a program, one can control which segments of code are run on each device through the casting operations. Each casting operation to and from the GPU pushes or pulls data back and forth from CPU memory to GPU memory. Excessive memory transfers should be avoided as they will reduce the performance of an application. The Jacket software minimizes these memory transfers automatically in normal operation. However, care must be taken in implementing an algorithm. Fortunately, the LBM can be completely vectorized and therefore all computations can be carried out on the GPU. Transfers to CPU memory are only necessary for outputting solutions at desired intervals. Currently, a transfer to CPU is necessary for MATLAB plotting routines; however, due to the nature of the GPU, plots can be created through OpenGL.

5.3. Computational Aspects

The basic code to be parallelized on the GPU using Jacket is written in MATLAB M-Language and follows the same traditional practice of explicitly separating the collision and streaming operations. The solution algorithm has not changed. However, in order to take advantage of the GPU and MATLAB, the codes must be vectorized. Due to vectorization, the solution procedure focuses on three main steps: the calculation of local macroscopic variables from distribution functions, the collision step and the streaming step. Two copies of the distribution functions are used to allow the code to be vectorized. The vectorized version of the code is straightforward. The Jacket GPU engine makes translating the code on the GPU as simple as casting the variables to the GPU. From there, all calculations are performed on the GPU. Since the LBM is inherently parallel, there is no need to cast variables back to the CPU until the end of the simulation or when variables are written to file.

6. Numerical Experiments

6.1. GPU Performance

The parallel performance on the GPU is investigated in this section. A rectangular lake, 170 km × 60 km, with a flat bottom was used to simulate wind-driven circulation. The model domain was discretized into four different grid cell sizes for comparison. The numbers of columns, rows, and layers are: 171 × 61 × 10, 341 × 121 × 10, 681 × 241 × 10, and 1361×481×10 with cell sizes of 1000 m, 500 m, 250 m, and 125 m, respectively, and 10 layers. Each layer has an initial local water height of 1 m, such that initial water depth is 10 m. The initial flow velocity is zero. We used τ = 0.501 and c = 20 m/s with ∆t calculated as ∆x/c for LBM parameters. Other parameters are f_c = 0 s⁻¹, ρ = 1025 kg/m³, ρ_a = 1.2 kg/m³, C_W = 0.0015, μ = 0.01 m²/s, $\kappa$ 0.001 m/s, U_Wx =7.4536 m/s and U_Wy = 0 (positive x wind direction). According to Equation (6), wind stress is ${\tau }_x^W$ = 0.1 N/m² along the x direction.

The EDFs in Equation (19) with static water and 1-m initial local water height determines the initial condition for the distribution functions. We applied free-slip bounce-back boundary conditions to the vertical walls.

The parallel performance of the GPU is based on arithmetic intensity and data access patterns [54]; therefore, the parallel performance is investigated based on the scalability of the speedup with increasing problem size. The simulations were run for a simulation time of 30 h where steady state has been achieved. The average time per time step is investigated to make a fair comparison on computational effort.

The execution time per time step and speedup for the GPU over a single core of the CPU in MATLAB for the four cases are shown in

Table 1. The grid size, execution time per time step and speedup for the graphics processing unit (GPU) over a single core of the central processing unit (CPU) in MATLAB.

Grid Size	CPU		Speedup
	Execution Time per Time Step (sec)
171 × 61 × 10	0.44	0.19	2.2
341 × 121 × 10	3.04	0.30	10.1
681 × 241 × 10	14.19	0.95	14.9
1361 × 481× 10	56.60	2.57	22.0

. It demonstrates the importance of arithmetic intensity in LBM performance on the GPU. If the number of lattice nodes is sufficiently high, the computations outweigh the data transferring overhead, yielding a high arithmetic intensity. The multilayer LB algorithm yields a speedup of approximately 2.2-fold on the smallest number of lattice nodes with the maximum speedup of approximately 22.0-fold on the largest number of lattice nodes.

6.2. Verification

We first verify the multilayer MRT-LBM using the steady-state analytical solutions in [52] by neglecting the effects of inertial terms and the unsteady terms for cases of wind-driven, density-driven and combined wind-driven and density-driven circulations. The velocity in the x direction is

$$u_x=g\frac{\partial H}{\partial x}\left\{\frac{z^2-H_0^2}{2\mu }-\frac{H_0}{\kappa }\right\}-\frac{g}{\rho }\frac{\partial \rho }{\partial x}\left\{\frac{z^3+H_0^3}{6\mu }+\frac{H_0^2}{2\kappa }\right\}+\frac{{\tau }_x^W}{\rho }\left\{\frac{z+H_0}{\mu }+\frac{1}{\kappa }\right\}$$

(32)

where H₀ is the still water depth. The free-surface slope term $g\partial H/\partial x$ is

$$g\frac{\partial H}{\partial x}=\frac{-\frac{g}{\rho }\frac{\partial \rho }{\partial x}\left\{\frac{H_0^4}{8\mu }+\frac{H_0^3}{2\kappa }\right\}+\frac{{\tau }_x^W}{\rho }\left\{\frac{H_0^2}{2\mu }+\frac{H_0}{\kappa }\right\}}{\left\{\frac{H_0^3}{3\mu }+\frac{H_0^2}{\kappa }\right\}}$$

(33)

The model is a 3400 m × 1400 m rectangular lake with a flat bottom. The initial water depth is 65 m. Three numerical models with difference number of layers were developed for comparison. The model domain was discretized into 501 × 206 lattices in the planar direction with a cell size of 6.8 m and five, ten and twenty layers. The corresponding initial local water heights for each layer is 13, 6.5, and 3.25 m, respectively. Fluid density is considered to be constant in this study but can have a constant density gradient along the longitudinal direction when density-driven circulation is considered. The LBM parameters are ∆t = 0.17 s and c = 40 m/s. To achieve a kinematic viscosity of $\nu$ = 1 × 10⁻⁶ m²/s, the relaxation time parameter in the SRT-LBM is given by $\tau =\frac{1}{2}+3\nu /c\Delta x$, i.e., τ = 0.5 + 1.1029 × 10⁻⁸. For the MRT-LBM, the relaxation rates s₄=s₆ =s₇=s₈=1/τ, and s₁=s₂ =s₇−0.6 were used. The linear friction law was used for bed friction. The approach to initialize distribution functions and the boundary conditions to the four vertical walls are the same as in the previous numerical case.

The wind-driven circulation validation is performed for two different wind stress values, ${\tau }_{iz}^W$ = 0.03 N/m² and ${\tau }_i^W$ = 0.3 N/m². Wind direction is along the positive x direction. The physical parameters for this case are ∂ρ/∂x = 0, ∂ρ/∂y = 0, ρ = 1025 kg/m³, ρ_a = 1.2 kg/m³, C_W = 0.0015, μ = 0.004 m²/s, $\kappa$ = 0.005 m/s. As shown in Figure 2, the multilayer MRT-LBM solutions from the 5-layer model to the 20-layer model compare well to the analytical solutions for u_x profile at the location (x, y) = (1700 m, 700 m) (the center of the lake). Figure 2 indicates that the 10-layer model is sufficient to capture the velocity profile.

The density-driven circulation validation is performed for two different horizontal density gradients, ∂ρ/∂x = −5 × 10⁻⁷ kg/m⁴ and ∂ρ/∂x = −5×10⁻⁵ kg/m⁴. The physical parameters for this case are ${\tau }_i^W$ = 0, ∂ρ/∂y = 0, ρ = 1025 kg/m³, ρ_a =1.2 kg/m³, C_W = 0.0015, μ = 0.004 m²/s, $\kappa$ = 0.005 m/s. The multilayer MRT-LBM solutions with three different numbers of layers compare well to the analytical solutions for u_x profiles for constant horizontal density as shown in Figure 3. Figure 3 indicates that the 10-layer model is sufficient to capture the velocity profile.

The multilayer MRT-LBM is also validated for combinations of wind-driven and density-driven circulation with ${\tau }_i^W$ = 0.03 N/m², ∂ρ/∂x = −5 × 10⁻⁵ kg/m⁴ and ${\tau }_i^W$ = 0.3 N/m², and ∂ρ/∂x = −5×10⁻⁴ kg/m⁴. The physical parameters for this case are ∂ρ/∂y = 0, ρ = 1025 kg/m³, ρ_a = 1.2 kg/m³, C_W = 0.0015, μ = 0.004 m²/s, and $\kappa$ = 0.005 m/s. With the positive x wind direction, as shown in Figure 4, the multilayer MRT-LBM solutions compare well to the analytical solutions for $u_x$ profile for the combined effects. Figure 4 indicates that the 10-layer model is sufficient to capture the velocity profile.

6.3. Wind-Driven and Density-Driven Circulation in Rotating Basins

In this example, the multilayer MRT-LBM is demonstrated on GPU-based HPC. The study simulates wind-driven, density-driven, and combined wind-density-driven circulations over a Gaussian bathymetry profile [39]:

$$H_0(y)=8+8exp\left[-{\left(\frac{y-3D/10}{2000}\right)}^2\right]+12exp\left[-{\left(\frac{y+3D/10}{2000}\right)}^2\right]$$

(34)

where $D$ is the width of the basin. The x axis coincides with the southern lateral wall of the basin and is pointed toward the head of the system. The y axis is laid along the closed boundary at x = 0. The numerical domain is 100 by 15 km. The maximum depth, 20 m, occurs at y = −3D/10 km. The local maximum depth, 16 m, occurs at y = 3D/10 km.

For each case, the grid size is 801 × 121 × 10. The LBM parameters are ∆x = 125 m, ∆t = 6.25 s, and c = 20 m/s. To achieve a kinematic viscosity of $\nu$ = 1 × 10⁻⁶ m²/s, the relaxation time parameter in the SRT-LBM is τ = 0.5 + 1.2 × 10⁻⁹. For the MRT-LBM, the relaxation rates, s₄ = s₆ = s₇ = s₈ = 1/τ, and s₁ = s₂ = s₇ − 0.6, were used. All closed boundaries have the free-slip bounce-back boundary condition. The initial water is stationary. The horizontal density gradient is constant. Uniform wind stress was linearly increased for the first six simulated hours and became constant after that. Wind direction is along positive x direction. We executed numerical simulations on a single workstation with a 3.0-GHz Intel^® Core™2 Extreme quad core processor and a NVIDIA^® Tesla™ C1060 Computing Processor. The parallel performance of a single core of the quad core processor and the Tesla are compared.

For the wind-driven case, the physical parameters are ${\tau }_i^W$ = 0.04 N/m², ∂ρ/∂x = 0, ∂ρ/∂y = 0, ρ = 1025 kg/m³, ρ_a = 1.2 kg/m³, C_W = 0.0015, μ = 0.004 m²/s, $\kappa$ = 0.0025 m/s, and $f_c$ = 1×10⁻⁴ s⁻¹. The wind velocity is U_Wx = 4.7140 m/s and U_Wy = 0. Figure 5a,b show the u_x distributions at plane x = 50 km and plane x = 98 km. The u_x is in the same direction as wind at all shallow depths along the transverse boundaries and the center of the channel. However, u_x is in the opposite direction of wind at deep depths. Figure 5c,d show the contours of the transverse flow u_y at plane x = 50 km and plant x = 98 km. The Coriolis effect produces surface elevation contours with strong lateral variability.

For the density-driven case, the physical parameters are ${\tau }_i^W$ = 0, ∂ρ/∂x = −5 × 10⁻⁸ kg/m⁴, ∂ρ/∂y = 0, ρ = 1025 kg/m³, ρ_a = 1.2 kg/m³, C_W = 0.0015, μ = 0.004 m²/s, $\kappa$ = 0.005 m/s, and $f_c$ = 1 × 10⁻⁴ s⁻¹. Figure 6 shows the u_x and u_y distributions at plane x = 50 km and plane x = 98 km. The u_x flows in the direction of the horizontal gradient at all depths in the shallow regions along the transverse boundaries and the center of the channel shown in Figure 6a,b. The u_x is in the opposite direction of horizontal gradient at deep depths. These flow directions are the opposite of those found in the purely wind-driven case. Highest flow occurs near the surface and flow decreases with depth due to the bottom friction. Figure 6c,d show the transverse flow u_y at plane x = 50 km and plane x = 98 km. Although the flow field is reversed for this case, the effect of the Earth’s rotation is consistent with the wind-driven case. Moreover, the transverse velocities exhibit similar behavior as in the wind-driven case with stronger magnitude at the x = 98 km plane. However, at the x = 50 km plane, the velocities are small, yet exhibit a vertical distribution of positive and negative flows.

For the combined wind-driven and density-driven case, the physical parameters are ${\tau }_i^W$ = 0.04 N/m², ∂ρ/∂x = −5 × 10⁻⁸ kg/m⁴, ∂ρ/∂y = 0, ρ = 1025 kg/m³, ρ_a = 1.2 kg/m³, C_W = 0.0015, μ = 0.004 m²/s, $\kappa$ = 0.005 m/s, and $f_c$ = 1 × 10⁻⁴ s⁻¹. Figure 7 shows the u_x and u_y distributions at plane x = 50 km and plane x = 98 km. For the combined wind-driven and density-driven case, the u_x distribution is similar to the density-driven case in terms of direction of the flow in shallow and deep regions as shown in Figure 7a,b. Figure 7c,d show the contours of the transverse flow, u_y at x = 50 km and x = 98 km planes. The flow patterns created by bottom friction, the Earth’s rotation, and bathymetry are all consistent with previous results. The main difference in the combined case is that the magnitudes of the flow are smallest near bed, increase in the positive z direction, and then decrease again near the surface. This is due to the bottom friction and the wind stress occurring in the opposite direction of the density gradient. The density gradient accounts for the direction of the flow, while the wind stress accounts for the smaller magnitude velocities near the surface.

The simulation time for each case was 47.3 h on a single core of the CPU and 1.68 h on the Tesla GPU, demonstrating a 28.16-fold speedup.

7. Conclusions

The lattice Boltzmann model (LBM) with multiple-relaxation-time (MRT) collision operation is a potential numerical method to simulate shallow water flow. The MRT-LBM can handle very low kinematic viscosity by using the last two relaxation rates (reciprocal of relaxation time) in D2Q9 lattice. Other relaxation rates can be determined with flexibility to ensure solution stability and accuracy. The MRT-LBM can avoid a stability problem which is often encountered when LBM with single-relaxation-time collision operation is used and the relaxation time is very close to 0.5.

The multilayer MRT-LBM is able to solve the multilayer Saint-Venant equations to obtain horizontal flow velocity distributions at different depths. The implementation of the multilayer MRT-LBM along with a given initial condition, boundary conditions, and forcing terms is straightforward. This study demonstrated the MRT-LBM to irregular bathymetry. The MRT-LBM solutions compare well to analytical solutions for horizontal velocity in various depths under the effects of wind-driven forcing, density-driven forcing, and their combined forcing.

The multilayer MRT-LBM is suitable for graphics processing unit (GPU) computing due to the locality of particle interaction and the transport of particle information in the LBM algorithm. For small grid sizes, the speedup is not impressive because the data transferring overhead between grid blocks on the GPU is not small compared to the actual computational cost. However, as the grid size increases, the computational cost becomes larger and dominates the data transferring overhead. This results in large speedup.