Simulation of Fire with a Gas Kinetic Scheme on Distributed GPGPU Architectures

Abstract

The simulation of fire is a challenging task due to its occurrence on multiple space-time scales and the non-linear interaction of multiple physical processes. Current state-of-the-art software such as the Fire Dynamics Simulator (FDS) implements most of the required physics, yet a significant drawback of this implementation is its limited scalability on modern massively parallel hardware. The current paper presents a massively parallel implementation of a Gas Kinetic Scheme (GKS) on General Purpose Graphics Processing Units (GPGPUs) as a potential alternative modeling and simulation approach. The implementation is validated for turbulent natural convection against experimental data. Subsequently, it is validated for two simulations of fire plumes, including a small-scale table top setup and a fire on the scale of a few meters. We show that the present GKS achieves comparable accuracy to the results obtained by FDS. Yet, due to the parallel efficiency on dedicated hardware, our GKS implementation delivers a reduction of wall-clock times of more than an order of magnitude. This paper demonstrates the potential of explicit local schemes in massively parallel environments for the simulation of fire.

1. Introduction

Fire is typically associated with a form of combustion that emits sufficient amounts of light and energy to be perceptible and self-sustaining on a macroscopic scale [1]. It has been known to mankind for ages, and for a million years, it has been used for different practical purposes [2]. Still, fire poses a great danger to the lives and well-being of humans. Building fires, such as the Grenfell tower disaster of 2017 in London, U.K., that caused the death of 72 people [3], show that fire safety is an important topic in building design, e.g., [4,5,6]. Fire simulation can be used in multiple ways to improve fire safety especially in the building design phase and in modeling based prediction of catastrophe scenarios. In the example of the Grenfell tower disaster, fire simulation was used a posteriori to investigate the cause of the catastrophe [7]. Other examples of a posteriori fire investigations can be found for example in [8,9,10,11].

The current standard in fire simulation is the Fire Dynamics Simulator (FDS), which is freely accessible and well documented [12,13,14,15]. It is based on a thermal compressible flow solver utilizing a low Mach number approximation. By this approximation, pressure perturbations travel infinitely fast, such that sound waves need not be resolved in the time domain, potentially permitting larger time steps. On the numerical side, this requires the solution of an elliptic pressure equation in every time step. Yet, the elliptic nature of this problem potentially impedes massive parallelization for most solvers available for this problem class. Another CFD code for the simulation of fire is the OpenFOAM [16] extension FireFOAM [17]. In the civil engineering community, also simpler zone models for the prediction of compartment fires are commonly used [18,19]. These models do not resolve the actual flame, but model the time evolution of integral variables in multiple zones and in multiple compartments.

Modern high performance kinetic flow solvers, such as the Lattice Boltzmann Method (LBM) [20,21,22,23], are utilized to compute approximate solutions of the compressible or weakly compressible Navier–Stokes equations. Such explicit schemes tend to require a relatively small time step as compared to implicit methods, but usually require less operations per time step and allow for straight forward parallelization, resulting in near to optimal scaling beyond tens of thousands of compute cores [24,25,26]. A drawback of the LBM is that the full and efficient inclusion of temperature for modeling of thermal compressibility for non-Boussinesq flows is not straight forward. Gas Kinetic Schemes (GKS) are another class of kinetic solvers for computational fluid dynamics. They were originally developed as flow solvers for high Mach number flows with shocks on structured [27,28,29] and unstructured [30,31,32,33] grids with a high order of accuracy [34,35,36,37,38]. GKS for smooth flow fields at low speeds, i.e., without shocks, were investigated in [39,40,41,42,43,44,45,46]. This is the flow regime considered in this work. Additionally, the gas kinetic nature of GKS allows extensions towards rarefied flows and micro fluid dynamics at finite Knudsen numbers in the frameworks of unified GKS [47,48,49,50,51] and discrete unified GKS [52,53,54,55,56,57].

In this work, we present the application of GKS for low speed flows for the simulation of three-dimensional turbulent natural convection and two fire plumes on two different spatial scales under the framework of Large Eddy Simulation (LES). All three tests were validated against experimental data and compared to reference simulations. For high performance computing, massively parallel hardware in the sense of multi-GPGPU was utilized.

2. Gas Kinetic Scheme Solver

In classical fluid mechanics, the flow evolution is described in terms of the Navier–Stokes equations coupled to a continuity and energy equation. This set of non-linear partial differential equations models the conservation of mass, momentum, and energy. Over the last few years, kinetic methods for fluid mechanics gained substantial popularity. Being typically implemented as explicit schemes, such methods show excellent parallel scaling properties. Kinetic methods are based on gas kinetic theory, rather than continuum mechanics. Therein, the fluid state is encoded as a statistical description of (fictitious) particle ensembles. The particle distribution function $f(\overrightarrow{x},\overrightarrow{u},t)$ is the probability to find a particle with velocity $\overrightarrow{u}$ at location $\overrightarrow{x}$ at time t. Hence, the probability is distributed not only in physical space, but also in velocity space. By considering actual particle motion and interactions, gas kinetic theory covers a more detailed physics than the macroscopic Navier–Stokes equations. The Navier–Stokes equations are in fact a limit of the gas kinetic theory in the case of a very small mean free path, i.e., vanishing Knudsen numbers and for the incompressible case of vanishing Mach numbers. The governing equation in gas kinetics is the Boltzmann equation (here with the Bhatnagar–Gross–Krook (BGK) collision operator [58]):

$$f_{,t}+\overrightarrow{u}·\nabla f={\displaystyle \frac{f^{eq}-f}{\tau }},$$

(1)

where $f^{eq}$ is the equilibrium distribution and $\tau$ a relaxation time. Inserting the first-order Chapman–Enskog expansion:

$$f\approx f^{eq}-\tau (f_{,t}^{eq}+\overrightarrow{u}·\nabla f^{eq})$$

(2)

and computing moments in velocity space yield the compressible Navier–Stokes equations.

In order to formulate a numerical scheme for the simulation of macroscopic fluid dynamics, a formal solution of the BGK-Boltzmann equation along particle trajectories is employed [27]. The solution along particle trajectories is approximated by Taylor expansion in space and time, and the initial non-equilibrium distribution is approximated as described in detail in Equation (2). Utilizing the first-order Chapman–Enskog expansion limits the validity of the resulting numerical method to physics described by the Navier–Stokes equations. Fluxes are then computed from the approximated time dependent solution for f.

In the present paper, we consider a simplification for smooth low-Mach number flows of the original GKS [39,45,46]. This simplified scheme has been validated in two dimensions for thermal compressible flows in the setting of natural convection on structured and unstructured grids [45]. Furthermore, a two-dimensional refinement algorithm for quad-tree-type non-uniform Cartesian grids was proposed for efficient grid generation and massively parallel implementation [46]. Therein, the GKS was implemented for GPGPUs utilizing the NVIDIA CUDA frame work [59], and high computational performance was shown. On GPGPUs with thousands of cores accessing the same memory, computational performance is often limited by memory access rather than by floating point operations [60]. The limiting factor in memory access can either be bandwidth or latency. The observation that the two-dimensional GKS utilizes up to 80% of the sustainable bandwidth [46] implies that the GKS algorithm is mostly memory-bound.

The code used in the present paper is a three-dimensional extension of the code used in [46]. The numerical kernel is essentially a straight forward extension to three dimensions, such that the reader is referred to [46] for further details. The same holds for the non-uniform grid implementation, where the procedure from [46] is implemented in three dimensions. The interface ghost cell interpolation coefficients for the second-order accurate interpolation are the same in 3D as in 2D. Additionally, the present code features an explicit sub-grid scale model for LES in terms of a static Smagorinsky eddy viscosity model [61].

A reduced version of the code is available online (with registration) under an open source license [62]. This preliminary version contains no spatial refinement and no combustion model, yet.

3. Performance of the Single GPGPU Implementation

All simulations in this work were run in single precision. The present implementation achieved update rates of up to 300 million cell updates per second for simulation of non-reacting flows on an NVIDIA Tesla P100 GPGPU in single precision. Including the fire module reduced the performance to about 215 million cell updates per second.

The utilization of the hardware for GPGPUs can be measured in terms of bandwidth utilization, because GPGPU computations are often memory limited, i.e., the time to read the data from the memory has a larger effect on the performance than the number of floating point operations. The theoretical peak bandwidth of the NVIDIA Tesla P100 is 732 GB/s. The maximal sustainable bandwidth was measured by the GPU-Stream tool [63]. For the NVIDIA Tesla P100 GPGPUs, the maximal sustainable bandwidth was found to be 543 GB/s. In order to estimate the memory traffic of the algorithm, we counted the number of bytes moved for a single cell update. For non-reacting flow, this was 1041 B. Our code had a net memory traffic of 312 GB/s, which was 57% of the sustainable bandwidth. The actual data traffic was reduced by the utilization of the cache of the GPGPU, which cannot be controlled directly. To gain an estimate, we omitted all computations from the code and performed only read and write operations. With this, we obtained an update rate of 825 GB/s, which was 152% of the sustainable bandwidth. We assumed here that the $100/152\times 1041$ B $\approx 685$ B was the actual number of bytes moved for one cell update and that the remaining 356 B were read from the cache. The corrected memory traffic accounted for 206 GB/s, which was 38% of the sustainable bandwidth. An ideal memory bound algorithm would hide all computations with memory traffic. However, for this to be possible, much more threads than cores have to be activated at the same time in the GPGPU. These threads have to allocate their own registers, such that their state persists even though the thread is currently not being processed. This allows for the GPGPU to switch between computation contexts, while threads have to wait for data due to memory latency. This mechanism is called latency hiding. A limiting factor for the number of active threads is the number of registers required per thread. The ratio of the actual number of active threads and the maximum number of active threads supported by the hardware is called occupancy. The occupancy is hence a metric for how well the GPGPU can hide the memory latency. The streaming multi-processors of the P100 support up to 64 warps of 32 threads and provide 65,536 registers. The flux computation kernel in the present GKS required 246 registers. Hence, eight warps could be active at the same time, which resulted in an occupancy of $8/64=12.5\%$. The actual achieved occupancy for the present implementation was measured with the CUDA profiling tools. The nvprof tool reported a measured occupancy of $12.3\%$ for the present implementation. This value indicates that latency hiding could not be utilized to its full extent.

4. Multi-GPGPU Implementation

Even though GPGPUs allow high performances for moderate system sizes, their memory is limited. The present paper features multi-GPGPU communication to overcome this limitation. The GKS algorithm is comprised of only local support, i.e., the flux computation requires only the next neighbor information. For multi-GPGPU, the flow domain is split into box-shaped sub-domains, where each sub-domain is processed by one GPGPU. For complete communication, it is sufficient to exchange a single layer of cells between neighboring sub-domains. The last layer of cells in the sub-domain is sent to the neighboring process, and one layer of communication ghost cells outside the sub-domain is received from the same neighboring process.

The communication procedure is shown in Figure 1. The flow state data of the exchanged cells is not continuous in memory. Therefore, as a first step, the flow state data of the sending cells is collected in a send buffer by a kernel. This send buffer is then sent to the neighboring process via Message Passing Interface (MPI) library calls. There, the data are received in a receive buffer and subsequently distributed to the communication ghost cells. These ghost cells are located at the same location as the corresponding send cells in the physical space of the simulation domain, but are directly outside the receiving sub-domain. Furthermore, the send and receive buffers must be downloaded and uploaded, respectively, because MPI can only access host memory. The host is the computer containing the GPGPU. Some modern MPI implementations feature CUDA support, where the network adapter can directly access GPGPU memory to improve communication performance. This procedure was implemented, but not used for the results in this work.

In three dimensions, one sub-domain can have a total of 26 neighboring sub-domains, connected via faces (6), edges (12), and corners (8). In order to reduce the number of communication neighbors, sequential communication along the coordinate axes is used; see Figure 1. This effectively reduces the number of communication neighbors to the six face neighbors. Sequential communication means that all communication in x-direction is completed on one process, before communication in the y-direction is started on this process. Similarly, the z-communication must only start after the y-communication is finished. In this way, information is also passed over edges and corners; see the highlighted cell in Figure 1.

Communication is a severe bottleneck in distributed computations. For GPGPUs, this effect is not as prominent as for distributed CPU communications, because the problem size per sub-domain is usually large compared to the message length of the communication. Nevertheless, communication invokes performance losses. The CUDA framework offers remedies for this. On CUDA-enabled GPGPUs, the compute hardware and the memory management hardware are mostly independent. The compute hardware can process data while memory is transferred between GPGPU and host memory. Further, compute kernel invocations are non-blocking with respect to the host, such that the host side communication (invocation of MPI routines) can also be done while the GPGPU processes data. This allows for communication hiding, which aims at performing computation and communication concurrently. Communication hiding is state-of-the-art in distributed GPU computing. Examples for communication hiding in LBM can be found, e.g., in [64,65,66].

The flux computation on cell faces in the vicinity of the sub-domain boundaries depends on the received cells. In order to keep the code logically consistent, the set of cell faces is divided into two sub-sets. The first contains all cell faces that were independent of the communication and the second faces that depend on the communication. The second set can only be processed after the communication is complete. Communication hiding is implemented in terms of CUDA streams. CUDA streams are independent queues, where each queue processes its tasks sequentially. One stream is used for the evaluation of the first sub-set of cell faces; see Stream 1 in Figure 2. This kernel is invoked first. The second stream handles all communication and interacts with the CPU; see Stream 2 in Figure 2. It finally computes the fluxes of the second sub-set of cell faces. The independent execution of the two streams is synchronized after the flux computation, and finally, all cells are updated. Figure 2 accounts for a single time step with communication in only one direction.

The algorithm for a single time step with communication hiding is given in Algorithm 1. This algorithm is an extension of the nested time stepping algorithm for GKS on a single GPU published in [46]. We utilize recursive time stepping to implement the nested time steps. The time step function is called with a level and a time step length. Each time step starts with the fine to coarse interpolation, if the current level is not the finest one. Subsequently, boundary ghost cells are set to fulfill the boundary conditions. Then, all faces that do not depend on communication on this level are processed. This is invoked in a non-blocking manner on the compute stream (Stream 1). Concurrently, the communication is performed on the communication stream (Stream 2). Here, we see the sequential communication per coordinate direction. When the communication is finished, the remaining faces are processed also on the communication stream (Stream 2). Afterwards, the streams are joined, by synchronizing the device, i.e., waiting for all streams to finish. If this level is not the finest, the coarse to fine interpolation is performed, and the nested time step function itself is called twice for the next finer level with half the time step length. This is the essence of nested time stepping. Performing two shorter time steps on the finer level saves time step evaluations on coarser levels by satisfying a local stability criterion instead of a global one. At the end of each time step, the cells are updated, i.e., the fluxes that were collected in an update variable are added to the current flow state. This function is repeated for the length of the simulation.

Algorithm 1 Recursive nested time stepping for multi-GPU.

1:  function nestedTimeStep($level,\Delta t$) 2:    3:      if $level\ne finestLevel$ then 4:          call fineToCoarseInterpolation($level$) 5:      end if 6:    7:      call setBoundaryConditions($level$) 8:    9:      call on Stream 1 computeFluxes($level$, $regularFaces$) 10:    11:      call on Stream 2 sendData($level$, x) 12:      call on Stream 2 recvData($level$, x) 13:    14:      call on Stream 2 sendData($level$, y) 15:      call on Stream 2 recvData($level$, y) 16:    17:      call on Stream 2 sendData($level$, z) 18:      call on Stream 2 recvData($level$, z) 19:    20:      call on Stream 2 computeFluxes($level$, $commFaces$) 21:    22:      synchronize device 23:    24:      if $level\ne finestLevel$ then 25:          call coarseToFineInterpolation($level$) 26:    27:          call nestedTimeStep($level+1,\Delta t/2$) 28:          callnestedTimeStep($level+1,\Delta t/2$) 29:      end if 30:    31:    32:      call updateCells($level$) 33:  end function

The performance of the multi-GPGPU implementation was measured in terms of parallel efficiency. The parallel efficiency is the ratio of the run times of coupled and independent sub-domains, i.e., with and without communication. The scaling was observed from a single GPGPU (as baseline) up to 32 GPGPUs. The baseline grid was a cube with uniform resolution. Under weak scaling, the workload per GPGPU is kept constant, such that the base cube was multiplied by the number of GPGPUs. Under strong scaling, the total workload is kept constant, such that the base cube was split into as many parts as there were GPGPUs. The decomposition with two GPGPUs was one-dimensional ($2\times 1\times 1$), with four GPGPUs two-dimensional ($2\times 2\times 1$), and with eight and more GPGPUs three-dimensional ($2\times 2\times 2$, $4\times 2\times 2$, and $4\times 4\times 2$). All tests were performed on the PHOENIX (https://www.tu-braunschweig.de/en/it/dienste/21/phoenix) compute cluster at TU Braunschweig, Germany. The cluster provides eight GPU nodes. Each is equipped with two INTEL Xeon E5-2640v4 CPUs, 64 GB of RAM, and four NVIDIA Tesla P100 GPUs with 16 GB HBM2 memory each. The nodes are connected by an Intel Omni-Path high speed network.

For weak scaling, the communication hiding was able to keep the parallel efficiency above $90\%$ for all tested problem sizes; see Figure 3a. Without communication hiding, the performance deteriorated at 32 GPGPUs, which was related to the hierarchical network topology in the HPC-system utilized for these benchmarks. The first 20 GPGPUs were connected to the same switch, while the remaining 12 GPGPUs were connected to another switch, such that the communication had to go an additional step. The good performance under weak scaling was due to the constant favorable ratio of computation per communication. For strong scaling, this ratio successively deteriorated, which can be seen in Figure 3b. For the larger problems, the communication hiding was able to keep the efficiency above $85\%$. For the smallest problem size, parallel efficiency deteriorated also when communication hiding was utilized. This was due to the small number of cells ($32\times 32\times 64$) per GPGPU, such that the GPGPUs could not utilize their full potential due to reduced latency hiding.

This section shows that the present implementation can efficiently distribute the workload to multiple GPGPUs, given that the total problem size is large enough.

5. Combustion Model

The GKS flow solver is augmented with a simple combustion model to demonstrate its applicability to simulate fire. The basic GKS tracks a single component fluid. For the simulation of fire, the local composition of this fluid is required. In GKS, multiple components can be implemented by tracking partial densities [67]. As an alternative, the single density can be split into multiple components based on transported scalars. In this paper, we follow the second approach. We use the scalar transport scheme for GKS proposed by Li et al. [68].

The fluid in the present model is composed of the three components, fresh air, gaseous fuel, and combustion products with mass fractions $Y_A$, $Y_F$, and $Y_P$, respectively. The two mass fractions of fuel and products are tracked explicitly in the simulation as scalar quantities, and the mass fraction of air is given implicitly by requiring that the sum of mass fractions is one. These three components are a simplification of the five species actually present in the reaction under consideration. Fresh air is comprised of a fixed mixture of oxygen, ${\mathrm{O}}_2$, and nitrogen, ${\mathrm{N}}_2$. The fuel is assumed to be pure methane, ${\mathrm{CH}}_4$. The reaction products are a fixed mixture of carbon dioxide, ${\mathrm{CO}}_2$, water, ${\mathrm{H}}_2\mathrm{O}$, and nitrogen, ${\mathrm{N}}_2$. The nitrogen is assumed to be inert. The stoichiometry of the one-step reaction is described by:

$${\mathrm{CH}}_4+2{\mathrm{O}}_2+n_{{\mathrm{N}}_2}{\mathrm{N}}_2⟶{\mathrm{CO}}_2+2{\mathrm{H}}_2\mathrm{O}+n_{{\mathrm{N}}_2}{\mathrm{N}}_2+\Delta H_{{\mathrm{CH}}_4},$$

(3)

where all chemical symbols are to be interpreted as one $\mathrm{mol}$ of the species and $n_{{\mathrm{N}}_2}$ is a stoichiometric factor of nitrogen that passively accompanies the reaction. The combustion of methane is an exothermic reaction with the molar heat of combustion $\Delta H_{{\mathrm{CH}}_4}$. In reduced species, the reaction is:

$$\Delta X_F+\Delta X_A⟶\Delta X_P+\Delta H_{{\mathrm{CH}}_4},$$

(4)

where $X_F$, $X_A$, and $X_P$ are the mole fractions of air, fuel, and products, respectively, and $\Delta$ denotes that only parts of the present amounts react. For the conversion of mole and mass fractions, see Appendix A. The reaction can similarly be written in terms of mass fractions with $h_{{\mathrm{CH}}_4}$ being the mass specific heat of combustion, such that:

$$\Delta Y_F+\Delta Y_A⟶\Delta Y_P+\Delta h_{{\mathrm{CH}}_4},$$

(5)

From the stoichiometry in the full reaction Equation (3), we see that $\Delta X_{{\mathrm{O}}_2}=2\Delta X_F$. In terms of mass fractions, the exact reaction mixture is given by the ratio:

$$s={\displaystyle \frac{\Delta Y_F}{\Delta Y_{{\mathrm{O}}_2}}}={\displaystyle \frac{\Delta X_F{M}_F/M}{2\Delta X_F{M}_{{\mathrm{O}}_2}/M}}={\displaystyle \frac{M_F}{2M_{{\mathrm{O}}_2}}},$$

(6)

where $M_{(·)}$ denotes molar masses. Finally, the reacting mass of fuel is defined by the minimum amount of fuel and oxygen as:

$$\Delta Y_F=min(Y_F,s{Y}_{{\mathrm{O}}_2}).$$

(7)

The mass of oxygen $Y_{{\mathrm{O}}_2}$ is computed from the mass of air by:

$$X_{{\mathrm{O}}_2}=X_{{\mathrm{O}}_2}^{\infty }{X}_A,\mathrm{such}\mathrm{that}{Y}_{{\mathrm{O}}_2}=\underset{{\displaystyle =Y_{{\mathrm{O}}_2}^{\infty }}}{\underbrace{X_{{\mathrm{O}}_2}^{\infty }{\displaystyle \frac{M_{{\mathrm{O}}_2}}{M_A}}}}{Y}_A,$$

(8)

where $X_{{\mathrm{O}}_2}^{\infty }$ is the mole fraction of oxygen in fresh air.

In more advanced models, the rate of the reaction is given by an Arrhenius-type reaction rate. In LES simulations such as discussed in this work, the reaction time scale is much shorter than a time step of the flow solver. In addition, a reaction can only take place where both fresh air and fuel are present, i.e., where they are mixed. Real mixing can only happen by molecular diffusion, which is usually slow. Turbulence can increase the diffusion by increasing the gradients and contact areas of the diffusing species. In LES, the small-scale turbulence is not resolved. Hence, the reaction is mixing limited, because reaction is fast and mixing is slow. In order to capture this behavior and the influence of the turbulence on the combustion, a model similar to the default model of Version 5 of FDS [69] is implemented.

The task of the combustion model is to predict the mass of reacting fuel $\Delta Y_F$. For this, the “mixed is burned” assumption is used [13]. In LES, the flow structures are usually not fully resolved, such that the flame-air interface thickness is much smaller than the cell size. Hence, a cell in the vicinity of the flame front will contain fuel, air, reacting fuel air mixture, and products. In this way, cells can be partially mixed. Thus, the assumption of instantaneous reaction in the mixed part of the cell is justified and adopted. The mixing time scale in the FDS 5 combustion model [69] used here is:

$$\tau ={\displaystyle \frac{0.1\Delta x^2}{D_{LES}}},$$

(9)

where $D_{LES}$ is the eddy diffusivity predicted by the sub-grid scale turbulence model. The mixed fraction of a cell is $\Delta t/\tau$, where $\Delta t$ is the time step of the flow simulation. The local heat release rate is then given by:

$${\dot{q}}^{‴}=\rho {\displaystyle \frac{min(Y_F,s{Y}_{{\mathrm{O}}_2})}{\tau }}\Delta h_{{\mathrm{CH}}_4}.$$

(10)

The amounts of consumed fuel, oxygen, and air over one time step are:

$$\Delta (\rho Y_F)={\displaystyle \frac{{\dot{q}}^{‴}}{\Delta h_{{\mathrm{CH}}_4}}}\Delta t,$$

(11)

$$\Delta (\rho Y_{{\mathrm{O}}_2})={\displaystyle \frac{\Delta (\rho Y_F)}{s}}$$

(12)

and

$$\Delta (\rho Y_A)={\displaystyle \frac{\Delta (\rho Y_{{\mathrm{O}}_2})}{Y_{{\mathrm{O}}_2}^{\infty }}}={\displaystyle \frac{\Delta (\rho Y_F)}{s{X}_{{\mathrm{O}}_2}^{\infty }}}={\displaystyle \frac{2M_A}{X_{{\mathrm{O}}_2}^{\infty }{M}_F}}\Delta (\rho Y_F),$$

(13)

respectively. Finally, the updates of mass fractions and energy due to the reaction are implemented as:

$$\begin{array}{cc}\hfill (\rho Y_F)(t+\Delta t)& =(\rho Y_F)(t)-\Delta (\rho Y_F)\hfill \end{array}$$

(14)

$$\begin{array}{cc}\hfill (\rho Y_P)(t+\Delta t)& =(\rho Y_P)(t)+\Delta (\rho Y_A)+\Delta (\rho Y_F)\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill (\rho E)(t+\Delta t)& =(\rho E)(t)+{\dot{q}}^{‴}\Delta t.\hfill \end{array}$$

(16)

Including this reactive model into the GKS solver induced additional stability issues. Accordingly, suitable limiters are designed to stabilize the simulations. At the leading fronts of flame plumes, excessive temperatures are observed. Thus, thermal diffusivity is locally increased in the presence of large temperature gradients as:

$$k_L=k+min\left(k_{L,max},C_T\Delta x^2\left|\right|\nabla T\left|\right|\right),$$

(17)

where $k_L$ is the increased thermal diffusivity, $k_{L,max}$ is an upper bound, and $C_T$ is an adjustable constant. This limiter smears out severe temperature peaks.

The scalar transport equations are subject to dispersion errors that produce negative and excessive mass fractions. Increased diffusivities are again used to repair this defects in a conservative fashion. The increased diffusivity $D_L$ for the generic passive scalar $\mathsf{\Theta }$ is:

$$D_L=\left\{\begin{array}{cc}D-C_{PS}\mathsf{\Theta }\hfill & \mathrm{for}\mathsf{\Theta }<0\hfill \\ D+C_{PS}(\mathsf{\Theta }-1)\hfill & \mathrm{for}\mathsf{\Theta }>1\hfill \\ D\hfill & \mathrm{else},\hfill \end{array}\right.$$

(18)

where $C_{PS}$ is an adjustable constant. In this work, $C_{PS}=0.1$ is chosen for all simulations.

Finally, the maximal local heat release is limited by an upper bound, i.e.:

$${\dot{q}}_L^{‴}=max\left({\dot{q}}^{‴},{\dot{q}}_{max}^{‴}\right),$$

(19)

where ${\dot{q}}_L^{‴}$ is the limited heat release rate and ${\dot{q}}_{max}^{‴}$ a maximal allowed heat release rate. Such a limiter is also used in Version 5 of FDS [69].

6. Validation

This section features three validation cases for the present flow solver. First, a simple setup of turbulent natural convection at a high Rayleigh number was simulated and compared to experimental data. Then, a small-scale and a large-scale fire were simulated and compared to experimental data to validate the combustion model. These results were also compared to the results obtained with Version 6 of FDS.

6.1. Turbulent Natural Convection

In our prior work on the development of GKS, we extensively validated GKS in two dimensions for various natural convection-related simulations, featuring validation based on reference simulations [45,46]. These included simulations of moderate Rayleigh numbers at high temperature differences and high Rayleigh numbers at small temperature differences due to the limited availability of reference data. In the present study, we present three-dimensional validation against experimental data. The availability of experimental data is, to the best of our knowledge, scarce. Ideally, we would seek a reference with a large temperature difference and a high Rayleigh number. The setup we feature here was investigated experimentally by Tian and Karayiannis [70,71] and numerically by Salinas-Vázquez et al. [72]. The setup includes a target Raleigh number of:

$$Ra={\displaystyle \frac{Prg{L}^3ϵ}{{\nu }^2}}=1.58\times {10}^9$$

(20)

and a dimensionless temperature difference of:

$$ϵ={\displaystyle \frac{\Delta T}{T_0}}=0.132,$$

(21)

where $T_0$ is the mean temperature. These parameters can be qualified as a high Rayleigh number, but at a small temperature difference compared to the $ϵ=1.2$ we used in the previous two-dimensional tests [45,46].

The experiment was performed in a square cavity with dimensions $L_x\times L_y\times L_z$ = 0.75 × 1.5 × 0.75 ${\mathrm{m}}^3$ [70]. The x-normal walls were kept at hot and cold temperatures, respectively. In the experiment, the wall temperatures were $50{}^{\circ }\mathrm{C}$ and $10{}^{\circ }\mathrm{C}$. Top and bottom walls, normal to the z-axis, were highly conducting, such that the experiment produced non-uniform temperature profiles along these walls. This is contrary to synthetic cavity flows with differentially heated sides, such as the benchmarks by de Vahl Davis [73] and Vierendeels et al. [74], where top and bottom walls were not insulated. In the simulations, a cubic fit of the measured data in the experiment was used. The fitted profiles are:

$${\displaystyle \frac{T_t-T_c}{\Delta T}}=0.65585-0.20368x-0.542x^2-2.7318x^3$$

(22)

and

$${\displaystyle \frac{T_b-T_c}{\Delta T}}=0.33714-0.26424x+0.53013x^2-2.6438x^3$$

(23)

with $x\in (-0.5,0.5)$ for top and bottom temperatures $T_t$ and $T_b$, respectively.

Further parameters for the simulation were a barometric number of $Ba=gL/(RT)=0.1$ (see [45] for the definition), a Prandtl number $Pr=0.71$, and $K=2$ internal degrees of freedom for the fluid molecules. The barometric number was set by the reference temperature and the Rayleigh number by viscosity. All remaining parameters were chosen as unity. The temperature dependent viscosity was modeled by Sutherland’s law; see, e.g., [75]. All velocity related results were normalized by a reference velocity [70]:

$$U_0=\sqrt{gL{\displaystyle \frac{\Delta T}{T_0}}}.$$

(24)

The gravitational acceleration g is implemented based on the so-called well-balanced scheme of Tian et al. [76].

The flow domain for the present simulation was a cube with dimension $L^3$ and with periodic boundary conditions in the y-direction to resemble the larger extent of the experimental cavity. The flow domain was discretized by a non-uniform grid with four grid levels and refinement towards the walls; see Figure 4a. The background grid had a resolution of ${128}^3$ such that the third refinement level had a wall resolution of $\Delta x/L=9.8\times {10}^{-4}$. This wall resolution was slightly coarser than the wall normal resolution in the reference simulation of Salinas-Vázquez et al. [72] where a stretched grid with $\Delta x/L=3\times {10}^{-4}$ at the wall was used. The grid was distributed to eight GPGPUs with a layout of $2\times 4\times 1$ subdomains.

As a first result, we show instantaneous turbulent structures as iso-surfaces of the Q-criterion [77]. It is apparent that the present simulation and the results of Salinas-Vázquez et al. [72] did not match in detail as the latter showed increased levels of turbulence on top and bottom walls.

For a more detailed validation, time averaged velocity profiles are compared. Vertical velocity profiles matched the experimental data well; see Figure 5a. The GKS slightly under-predicted the peak velocities on the top left and bottom right, while the reference simulation over-estimated these velocities. The horizontal velocities showed large discrepancies between all three datasets, such that no clear result could be gained; see Figure 5b. In the temperature profiles, the present GKS results showed deviations from the experimental data that were not present in the reference simulations; see Figure 6a. The GKS results were non-monotonic, i.e., they under-estimated temperature just outside the heated boundary layer and over-estimated temperature just outside the cooled boundary layer. Overall, the temperature field was captured well. The thermal boundary layer thicknesses were correctly estimated. Finally, the turbulence was measured as the root-mean-squared vertical velocity fluctuation. The reference simulation substantially over-estimated the turbulence on top left and bottom right, while the GKS essentially correctly predicted the strength and profile of the turbulence; see Figure 6b. This finding is consistent with the discrepancy in turbulent structures, observed in Figure 4b. It supports the interpretation that the present GKS results resembled the experiment better in that region than the reference simulation.

Overall, this simulation validates the usage of the present GKS for convection-driven turbulence.

6.2. Purdue Flame

The reference software for fire simulation FDS in its current Version 6 [12] has been validated for hundreds of test cases; see [15]. Two of these test cases are considered here. The first example is a table top experiment performed at Purdue University, USA, and published by Xin et al. [78]. The experiment featured a methane flame over a $7.1$ cm diffuser burner with an outlet velocity of $0.0314\mathrm{m}{\mathrm{s}}^{-1}$, a mass flow rate of $84.3\times {10}^{-6}\mathrm{k}\mathrm{g}{\mathrm{s}}^{-1}$, and hence, a density of $0.68\mathrm{k}\mathrm{g}{\mathrm{m}}^{-3}$. With the specific heat of combustion for methane of $\Delta h_{C{H}_4}=50\mathrm{M}\mathrm{J}{\mathrm{k}\mathrm{g}}^{-1}$, the total heat release rate of the flame is given by $\dot{q}={\dot{m}}_{C{H}_4}\Delta h_{C{H}_4}=4.2\mathrm{k}\mathrm{W}$.

This setup was computed with both FDS Version 6 and with the present GKS. The FDS input file for this test case is available in the FDS repository on GitHub [79]. The computational domain for GKS spaned $0.15\times 0.15\times 0.4{\mathrm{m}}^3$ with uniform spatial resolutions of $\Delta x=4$ cm and $\Delta x=2$ cm for a coarse and fine simulation, respectively. The lateral sides of the domain featured an open boundary condition that allowed inflow at ambient conditions if the internal pressure is below the ambient pressure and prohibits outflow. The top of the domain was a pressure outflow boundary that explicitly prohibits inflow. The bottom was an insulated wall, and the burner was modeled as a Neumann boundary for mass flux and a symmetry boundary for velocity and temperature.

The simulation was performed at physical parameters with density of air $\rho =1.2\mathrm{k}\mathrm{g}{\mathrm{m}}^{-3}$, gravitational acceleration $g=9.81\mathrm{m}{\mathrm{s}}^{-2}$, viscosity $\mu =1.8·{10}^{-5}$, $Pr=0.71$, and $K=2$. In order to accelerate the simulation, ambient temperature and heat of combustion were scaled down by a factor of 100. This substantially increased the Mach number and reduced the discrepancy between the time scales of the hydrodynamic and the acoustic modes.

The gravitational acceleration was implemented in a different way than in the previous example for these fire simulations. The acceleration only changed momentum in the cell while temperature was kept constant. The energy was then computed from the new momentum and the old temperature. The temperature and heat release rate limiters were effectively turned off for this test case by setting a low value for $C_T={10}^{-8}$ and a very large value for ${\dot{q}}_{max}^{‴}=500\mathrm{M}\mathrm{W}{\mathrm{m}}^{-3}$.

In this work, we aim at comparing two completely different numerical methods and implementations, which are, to a certain degree, able to simulate similar physical setups. The main difference between the present GKS and FDS is the basic flow solver. Apart from that, the present GKS utilizes a static Smagorinsky turbulence model, while FDS implementes a local version of Deardorff’s turbulence model [13,80]. Further, the present GKS employes a simple combustion model with a single reaction time scale that modeles the turbulent mixing. In FDS, this model is refined by taking into account multiple physical mixing processes that play a role in the computation of the mixing time scale. Therein, mixing on coarse scales is based on buoyant mixing, while the turbulent mixing time scale comes into play at intermediate scales. On small scales, diffusive mixing is modeled. Hence, this refined model takes into account the mixing mechanism present on the scale of the grid cells.

A detailed comparison between FDS and GKS was not possible because many aspects are implemented differently. Hence, we aim at a comparison of the final results, only. These are compiled as time averaged profiles of flow properties in addition to the absolute wall-clock time required for the simulation. The quality of these results can be assessed independently of how they were produced. This comparison does not give any insight into which part of the software triggered certain deviations between the results of both simulations, though.

For the sake of validation, time averaged profiles through the flame were collected at different heights as given in [78]. Two velocity components were considered here. The vertical velocity component along the vector of gravitational acceleration is called axial velocity W, because the flame should in theory show a rotational symmetry. The second component is the radial velocity. For profiles along the x-axis, the radial velocity corresponds to the horizontal velocity U (except of the sign). The circumferential velocity component was not considered here, because the flames were not rotating, and consequently, this velocity component was not measured during the experiment. Any present circumferential velocities in the simulations would be due to insufficient averaging.

During the initial comparison of the measured and simulated data, it became obvious that the experimental data were not centered properly. In order to correct for this, we shifted all experimental results for this test case by $\mathsf{\delta }x=0.0028\mathrm{m}$ in the negative x-direction. This value was obtained by computing the normalized first moment of the vertical velocity distribution, i.e.,

$$\mathsf{\delta }x={\displaystyle \frac{\sum x_i\overline{W}(x_i,z_i)}{\sum \overline{W}(x_i,z_i)}},$$

(25)

for all available data points in the experimental dataset. This procedure centered the experimental data and made comparison with the centered simulations consistent.

The axial velocities of both simulations show a reasonable agreement to the experimental data for both FDS and GKS; see Figure 7a. FDS tended to over-estimate the plume width, while GKS tended to underpredict the width, especially at higher resolution. Both simulations featured a velocity drop in the center of the flame that was not present in the experimental data. In terms of maximal velocity magnitude, FDS tended to over-estimate and GKS to under-estimate. The radial velocity profiles of both simulations showed the measured flow from the inside to the outside; see Figure 7b. The experimental data featured a strong peak in radial velocity at about $x=2$ cm. The strength of this peak was only captured by the coarse FDS simulation. Both the fine grid FDS and GKS simulation did not show such a strong peak.

The experimental temperature profiles are only reported for the positive x-axis. The absolute temperature magnitude was under-predicted by GKS, while FDS obtained a good agreement; see Figure 8a. In accordance with the axial velocity profiles, the fine GKS simulation under-predicted the flame width. FDS on the other hand captured nearly the correct flame width with a slight tendency to over-predict. The magnitude of the root-mean-squared vertical (axial) velocity fluctuations was captured with acceptable accuracy; see Figure 8b. The shape of the profiles deviated substantially from the experiment for all simulations. GKS showed less turbulence than FDS.

In addition to the profiles, we measured the quality of the simulations in terms of $L_2$ deviation from the experimental data. For the generic quantity $\varphi$, the deviation to ${\varphi }^{Exp}$ was computed as:

$$L_2={\displaystyle \frac{\sum {\left({\varphi }_i-{\varphi }_i^{Exp}\right)}^2}{\sum {\left({\varphi }_i^{Exp}\right)}^2}},$$

(26)

where the simulations results ${\varphi }_i$ are interpolated linearly towards the location of the experimental data. The sums therein were computed over all grid points on all heights. The $L_2$ deviations for velocities, temperature, and turbulent kinetic energy are shown in Figure 9. A prevalent accuracy between the two methods cannot be found. Furthermore, no substantial improvement in accuracy with refinement was observed for the two simulation methods. In this metric, the GKS results even deviated more from the experimental data with refinement. Nevertheless, the overall accuracy of both methods is comparable. Hence, these results show the general applicability of GKS for the simulation of fire, while some deficiencies in predicting the maximal flame temperature with respect to FDS were observed.

Finally, we compared the absolute times to the solution for both FDS and GKS; see Figure 10. FDS was run on a single CPU core, four cores, and twenty cores. All cores were in the same computer node, which was equipped with two ten-core INTEL Xeon E5-2640v4 CPUs. While using four cores brought a substantial speedup, using more cores brought no further improvement in this experiment. GKS was run on a single NVIDIA Tesla P100 GPGPU. While FDS required between several hours and multiple days to compute twenty seconds of real-time in the simulation, GKS terminated in about half an hour on the coarse grid and nine and a half hours for the fine grid. This resulted in a speedup of about 24 for the coarse and 16 for the fine simulation, demonstrating the capabilities of massively parallel implementations on GPGPUs. While one could argue that the FLOPS rate of the used CPU was substantially lower than of the GPGPU and one should rescale the corresponding wall-clock times accordingly, we note that the inherent FDS solvers cannot utilize the GPGPU architecture, and thus, the GKS has the advantage to benefit from this many-core architecture. In addition, the scalability of the GKS on massively parallel CPU-hardware bears no special challenges, but was not addressed in this work.

6.3. Sandia Flame

The second test case was experimentally investigated at Sandia National Laboratory, Albuquerque, New Mexico, USA, and published by Tieszen et al. [81,82]. This second test had a similar setup to the Purdue flame, but took place at a much larger scale. The burner now had a diameter of one meter. We considered only Test 24 of the Sandia flame experimental series. It was characterized by a total heat release rate of $2.08\mathrm{M}\mathrm{W}$ with a methane density of $0.5464\mathrm{k}\mathrm{g}{\mathrm{m}}^{-3}$ and a mass flow rate of $0.0416\mathrm{k}\mathrm{g}{\mathrm{s}}^{-1}$. The properties of air were the same as for the Purdue flame. The flow domain covered $3\times 3\times 4{\mathrm{m}}^3$. Three uniform discretizations with $\Delta x=6.25$ cm, $\Delta x=3.125$ cm and $\Delta x=1.5625$ cm were considered. For this test case, the temperature and heat release rate limiters were activated choosing $C_T={10}^{-3}$ and ${\dot{q}}_{max}^{‴}=2\mathrm{M}\mathrm{W}{\mathrm{m}}^{-3}$. Both limiters were used to stabilize the simulation. The value for $C_T$ was chosen by successively increasing the value, until the simulation remained stable. The value for ${\dot{q}}_{max}^{‴}$ was taken from FDS Version 5 [69], which also includes this limiter. The justification for this limit is physically based on a scaling analysis for pool fires, which suggested that the spatial average of the heat release rate of fires is about $1.2\mathrm{M}\mathrm{W}{\mathrm{m}}^{-3}$ [83].

The instantaneous temperature fields of both FDS and GKS simulations at the mid-planes for the highest resolution are shown in Figure 11. GKS appeared to be more dissipative than FDS, which is also confirmed below. The maximal temperatures were similar. FDS showed only a small dependence on the spatial resolution; see the time averaged vertical (axial) velocity profiles Figure 12a. FDS also captured the maximal velocity of the experiment well, but predicted a slightly broader plume. GKS, on the other hand, showed a larger deviation between the three spatial resolutions. At $z=0.3$ m, the experimental profile was matched almost exactly for the highest resolution. In the higher profiles, GKS tended towards a slightly thinner profile compared to the experimental data. In the radial velocity profiles, we found an even stronger dependence on the spatial resolution, but the profiles converged towards the experimental data; see Figure 12b. Temperature profiles were not reported in the experiment, such that only FDS and GKS are compared in Figure 13a. The FDS profiles of all three resolutions nearly coincided, while the GKS results showed a strong dependence on spatial resolution. The turbulent kinetic energy $TKE=(\overline{U^{\prime }{U}^{\prime }}+\overline{V^{\prime }{V}^{\prime }}+\overline{W^{\prime }{W}^{\prime }})/2$ was probably the most interesting profile when considering the finding in Figure 11. At a height of $10$ cm above the burner, GKS predicted near to no turbulence. Here, we saw a clear difference from the FDS results, which was consistent with the instantaneous temperature fields. The coarsest resolution remained at a very low level of turbulence at all heights. The higher resolutions showed more turbulence and seemed to converge towards the experimental data. FDS delivered the right level of turbulence for the three spatial resolutions, but the width of the plume was over-estimated.

Similar to the Purdue flame test case, we also present the $L_2$ deviations between simulations and experimental data; see Figure 14. For this test case, we saw a clear improvement with a reduced cell size for both methods. For GKS, the deviation on the coarse grid was larger than for FDS. Nevertheless, with refinement, GKS reached similar to or better agreement than the experimental data.

Furthermore, we investigated the radial symmetry of the axial (see Figure 15) and radial (see Figure 16) velocity fields. The velocity fields were not perfectly rotational symmetric.

Run times are shown in Figure 17. GKS was between 55 and 40 times faster than FDS. Overall, this test case shows the same tendency as the small-scale Purdue flame: GKS can obtain plausible results for these fire simulations. GKS shows a stronger dependence on the spatial resolution, but is substantially faster, compared to FDS.

7. Conclusions

In the present paper, we demonstrated the efficient usage of gas kinetic schemes for the simulation of fire plumes. To that end, a high performance implementation of GKS for low-Mach number flows in multi-GPGPU environments was presented. The validity of the solver was shown in two dimensions in prior work [45,46]. Here, we showed the extension to three dimensions with an update rate on a single GPGPU of 300 million cell updates per second. On multi-GPU architectures, we demonstrated a high parallel efficiency of over $85\%$ for up to 32 GPGPUs, enabled by communication hiding. The flow solver was augmented by a simple combustion model based on partial mixing and instantaneous reaction for the one-step methane combustion.

The present GKS was validated for three-dimensional, turbulent natural convection, where it showed good agreement with experimental data for first- and second-order statistics. The present fire model was tested on the simulation of two fire plume simulations, one on a small scale and one on a large scale. We presented validation against experimental data and comparison to the standard software FDS. Both tests showed consistent results: The overall magnitudes of temperature and velocity were captured well by the present GKS. For FDS, the results at different spatial resolutions nearly coincided. A limitation of the present approach was that GKS showed substantial deviations between the results for different spatial resolutions. However, the results converged to the experimental data. We assum that the strong dependence on spatial resolution is due to the simple turbulence and combustion models in the present GKS, while FDS features more elaborate models. Future investigations are required to investigate this. Even though the accuracy of the results is slightly lower, we demonstrated substantial speed-ups of more than an order of magnitude in time to solution with respect to FDS.

We conclude that, despite the presented shortcomings, GKS is a viable candidate for the simulation of fire. The focus of future research in this direction has to include advanced turbulence and combustion modeling.