A Parallel Algorithm Based on Regularized Lattice Boltzmann Method for Multi-Layer Grids

Abstract

The regularized lattice Boltzmann method (RLBM) is an improvement of the lattice Boltzmann method (LBM). The advantage of RLBM is improved accuracy without increasing computational overheads. The paper introduces the method of multi-layer grids, the multi-layer grids have different resolutions which can accurately solve problems in computational fluid dynamics (CFD) without destroying the parallelism of RLBM. Simulating fluid flow usually requires a large number of grid simulations. Therefore, it is necessary to design a parallel algorithm for RLBM based on multi-layer grids. In this paper, a load-balancing-based grid dividing algorithm and an MPI-based parallel algorithm for RLBM on multi-layer grids are proposed. The load balancing-based grid dividing algorithm ensures that the workload is evenly distributed across processes, minimizing the discrepancies in computational load. The MPI-based parallel algorithm for RLBM on multi-layer grids ensures accurate and efficient numerical simulation. Numerical simulations have verified that the proposed algorithms exhibit excellent performance in both 2D and 3D experiments, maintaining high stability and accuracy. The multi-layer grids method is significantly better than single-layer grids in terms of CPU runtime and number of grids required. Comparative analysis with the OpenMP multi-threading method on the multi-layer grid RLBM shows that the proposed algorithm in this paper achieves superior speedup and efficiency.

1. Introduction

The lattice Boltzmann method (LBM) as a mesoscopic scale computational fluid dynamics method [1] has been widely recognized as an effective means of dealing with complex fluid flows [2]. For example, simulating the fluid flow of an infinitely wide wedge [3], coupled conduction and radiative heat transfer in composite materials [4] or indoor airflow [5].

The advantages of LBM are easily handling complex boundaries [6] and high parallelism [7]. However, LBM faces some challenges such as numerical instability at high Reynolds numbers [8]. In order to solve this problem, the regularized lattice Boltzmann method (RLBM) was proposed [9]. The principle of RLBM is to perform a pre-collision step before the collision and streaming steps [10]. The pre-collision step restores symmetries that may not be inherently satisfied during numerical simulations [11]. RLBM maintains the high parallelism of the LBM without significantly increasing the amount of computation [12]. Latt and Chopard [9] introduced a regularisation model based on single relaxation time BGK dynamics. It was demonstrated that the RLBM reduces the computational cost and has improved accuracy and stability over the LBM. Malaspinas [13] proposed regularized boundary conditions for dealing with straight boundaries. The method is applied to standard grids (D2Q9 and D3Q19). Guo [14] discusses arbitrary surface boundary conditions for compressible flows in the context of regularized models. Otomo [15] applied regularized collision models to phase-field multiphase flow models. Basha and Nor Azwadi [16] showed the similarity between the structure of LBM and RLBM. The RLBM has the properties of easily handling complex boundary conditions and high parallelism [17].

RLBM typically uses a uniform grid for numerical calculations. The fact that the accuracy of the grid cannot be flexibly adjusted limits the application of RLBM in some fields. In numerical simulations, there are regions where flow details need to be finely captured. Refining the entire grid would significantly increase the computational overhead [18]. Therefore, it is necessary to design a method for local grid refinement. Filippova and Hanel [19] proposed an FH method where the information of the grid is stored in the vertices. The coarse and fine grids in the FH method exchange information in a bi-directional coupled way. An adaptive mesh refinement (AMR) is proposed, which effectively reduces the loss of computational accuracy [20]. Liu et al. [21] used the AMR method in combination with the immersion boundary lattice Boltzmann method (IB-LBM) to validate the effectiveness of the method in both 2D and 3D problems. Since the distribution functions are exchangeable on different grids, Lin and Lai [22] propose a single coupling method. The method is subject to errors in some cases. Dupuis and Chopard [23] proposed a bi-directional coupled local grid refinement method, which solves the error problem in an excellent way. The method can improve the computational efficiency. In addition, a buffer-based interpolation method between coarse and fine grids was proposed by Chen et al. [24]. The method effectively reduces the computation time. A method for storing grid information in the center point was proposed by Rohde et al. [25] The grid information at different sizes is stored in the grid intersections. Storing the grid information using this method can accurately and uniformly spread the grid information, which improves the accuracy of numerical calculations [26].

The RLBM with multi-layer grids uses a uniform grid for each layer, and the size of the previous grid is twice the size of the subsequent grid. In using the method to calculate equilibrium distribution functions, collision, streaming, and computing macroscopic quantities are localized. Therefore, excellent conditions are provided for the parallelization of RLBM. Hasert, M et al. [27] proposed a new parallel approach that is suitable for numerical simulations under large-scale clustering. A method of computational domain division by one and two dimensions was proposed by Kandhai et al. [28]. The method effectively improves the parallel performance and achieves accurate numerical results. Numerical experiments on multiphase flow were carried out by Pan et al. [29]. The performance differences between different domain divisions were analyzed. In engineering problems, parallel computation of LBM is also applied in different flow domains. Such as nuclear reactor fuel cooling [30] and vehicle external flow field [31]. A load-balancing method was proposed by Schornbaum et al. [32]. The method requires that all grids be distributed to all parallel domains. A parallel algorithm was designed by Abas et al. [33]. The algorithm is based on numerical experiments implemented on the Palabos library [34] with RLBM and MPI cross-node parallelism. Good results were achieved; however, the implementation details of the algorithm were not disclosed. Multi-layer grids RLBM is capable of handling large-scale computational CFD problems. Servers [35] and supercomputers [36] with multiple cores are common choices for solving problems in current scenarios.

These methods demonstrate that RLBM is a stable method. Local grid refinement of multi-layer grids can concentrate the parts that need to be computed accurately in more desired regions. RLBM with multi-layer grids is suitable for parallelization. Based on this, a load balancing-based grid dividing algorithm and an MPI-based parallel algorithm for RLBM on multi-layer grids are proposed in this paper. The load balancing-based grid dividing algorithm ensures that the workload is evenly distributed across processes, minimizing discrepancies in computational load. The MPI-based parallel algorithm for RLBM on multi-layer grids ensures accurate and efficient numerical simulation. The algorithm satisfies load-balanced grid division and MPI-based multi-processing parallel design.

The paper is structured as follows. The Section 1 is the introduction. Section 2 introduces the RLBM and gives the RLBM evolution process for multi-layer grids. Section 3 proposes load balancing-based grid dividing algorithm and MPI-based parallel algorithm for RLBM on multi-layer grids. Numerical experiments and parallel performance analysis are in Section 4. Conclusions are in Section 5.

2. RLBM Based on Multi-Layer Grids

2.1. Governing Equation for RLBM

The advantage of RLBM is that it can improve stability and accuracy without materially increasing computational effort. The idea of the regularized model is to introduce a pre-collision distribution function in front of the collision and streaming.

The evolution equation for the RLBM is as follows:

$$f_i(\mathit{r}+{\mathit{e}}_i{\delta }_t,t+{\delta }_t)=(1-{\displaystyle \frac{1}{\tau }})f_i^1(\mathit{r},t)+f_i^{eq}(\mathit{r},t)$$

(1)

where $f_i$ is the distribution function [37] at site $\mathit{r}$ and time t, $f_i^1$ is the first-order distribution function. ${\delta }_t$ is the lattice time step. $\tau$ is the dimensionless relaxation time, which is determined by the fluid viscosity of the fluid. ${\mathit{e}}_i$ is the discrete velocity.

The distribution function satisfies the following relationship:

$$f_i-f_i^{eq}=f_i^{\mathrm{neq}}$$

(2)

where $f_i^{eq}$ is the equilibrium distribution function and $f_i^{\mathrm{neq}}$ is the non-equilibrium distribution function. The explicit formula for $f_i^{\mathrm{neq}}$ is in Appendix A. $f_i^{\mathrm{neq}}$ is approximately equal to the $f_i^1$. The first-order distribution function $f_i^1$ has the following relationship:

$$f_i^{\mathrm{neq}}\approx f_i^1={\displaystyle \frac{{\omega }_i}{2c_s^4}}{\prod }_{\alpha \beta }^{\mathrm{neq}}{Q}_{i\alpha \beta }$$

(3)

where ${\prod }_{\alpha \beta }^{\mathrm{neq}}$ is the stress tensor. ${\prod }_{\alpha \beta }^{\mathrm{neq}}$ satisfies the following relationship:

$${\Pi }_{\alpha \beta }^{\mathrm{neq}}={\Pi }_{\alpha \beta }-\sum f_i^{eq}{\mathit{e}}_{i\alpha }{\mathit{e}}_{i\beta }$$

(4)

The equilibrium distribution function $f_i^{eq}$ can be obtained from the following equation:

$$f_i^{eq}=\rho w_i\left[{\displaystyle \frac{3}{c_s^2}}{\mathit{e}}_i·\mathit{u}+{\displaystyle \frac{9}{2c_s^4}}{({\mathit{e}}_i·\mathit{u})}^2-{\displaystyle \frac{3}{2c_s^2}}\mathit{u}·\mathit{u}+1\right]$$

(5)

where $\rho$ is the macroscopic density, $c_s$ is the lattice sound velocity, $c_{\mathrm{s}}=1/\sqrt{3}$. ${\omega }_i$ is the weighting factor of the corresponding grid direction, $\mathit{u}$ is the macroscopic velocity.

As shown in Figure 1, this paper uses the D2Q9 model to solve the two-dimensional problem.

Figure 1. (a) The D2Q9 velocity discrete model. (b) The D2Q9 distribution function model.

The weighting factor ${\omega }_i$ is represented as follows:

$${\omega }_i=\left\{\begin{array}{c}4/9,\hfill \\ 1/9,\hfill \\ 1/36,\hfill \end{array}\right.\begin{array}{c}\\ \\ \end{array}\begin{array}{c}\\ \\ \end{array}\begin{array}{c}\\ \\ \end{array}\begin{array}{c}i=0,\hfill \\ i=1,2,3,4,\hfill \\ i=5,6,7,8.\hfill \end{array}$$

(6)

The discrete velocity ${\mathit{e}}_i$ is represented as follows:

$${\mathit{e}}_i=\left[\begin{array}{c}{\mathit{e}}_0,{\mathit{e}}_1,\cdots ,{\mathit{e}}_8\end{array}\right]=\left[\begin{array}{ccccccccc}0& 1& 0& -1& 0& 1& -1& -1& 1\\ 0& 0& 1& 0& -1& 1& 1& -1& -1\end{array}\right]$$

(7)

As shown in Figure 2, this paper uses the D3Q19 model to solve the three-dimensional problem.

Figure 2. (a) The D3Q19 velocity discrete model. (b) The D3Q19 distribution function model

The weighting factor ${\omega }_i$ is represented as follows:

$${\omega }_i=\left\{\begin{array}{c}1/3,\hfill \\ 1/18,\hfill \\ 1/36,\hfill \end{array}\right.\begin{array}{c}\\ \\ \end{array}\begin{array}{c}\\ \\ \end{array}\begin{array}{c}\\ \\ \end{array}\begin{array}{c}i=0,\hfill \\ i=1,2,\cdots ,6,\hfill \\ i=7,8,\cdots ,18.\hfill \end{array}$$

(8)

The discrete velocity ${\mathit{e}}_i$ is represented as follows:

$${\mathit{e}}_i=\left[\begin{array}{ccccccccccccccccccc}0& 1& -1& 0& 0& 0& 0& 1& -1& 1& -1& 1& -1& 1& -1& 0& 0& 0& 0\\ 0& 0& 0& 1& -1& 0& 0& 1& -1& -1& 1& 0& 0& 0& 0& 1& -1& -1& 1\\ 0& 0& 0& 0& 0& 1& -1& 0& 0& 0& 0& 1& -1& -1& 1& 1& -1& 1& -1\end{array}\right]$$

(9)

The symmetric tensor $Q_{i\alpha \beta }$ is defined as follows:

$$Q_{i\alpha \beta }={\mathit{e}}_{i\alpha }{\mathit{e}}_{i\beta }-c_s^2{\delta }_{\alpha \beta }$$

(10)

In the RLBM model, the momentum flux tensor, the macroscopic velocity and the macroscopic density are given by

$${\Pi }_{\alpha \beta }=\sum _{i=0}{f}_i{\mathit{e}}_{i\alpha }{\mathit{e}}_{i\beta }$$

(11)

$$\rho \mathit{u}=\sum _{i=1}{\mathit{e}}_i{f}_i=\sum _{i=1}{\mathit{e}}_i{f}_i^{eq}$$

(12)

$$\rho =\sum _{i=0}{f}_i=\sum _{i=0}{f}_i^{eq}$$

(13)

The evolution equation consists of two parts:

Collision:

$$f_i^{+}(\mathit{r},t)=(1-{\displaystyle \frac{1}{\tau }})f_i^1(\mathit{r},t)+f_i^{eq}(\mathit{r},t)$$

(14)

Streaming:

$$f_i(\mathit{r}+{\mathit{e}}_i{\delta }_t,t+{\delta }_t)=f_i^{+}(\mathit{r},t)$$

(15)

In this paper, the boundary processing uses two formats: straight boundary [38] and YMS format [39]. In multi-layer grids, RLBM, the innermost grid surrounds the entire object. Therefore, it can be viewed as using a single-layer grid for curved boundaries using the YMS format. The YMS format can be directly applied to the multi-layer grids RLBM program.

The evolution of the RLBM is depicted in Figure 3.

Figure 3. Flow chart of RLBM.

2.2. RLBM Based on Multi-Layer Grids

To facilitate the description of the model, the 2D grid is depicted in Figure 4.

Figure 4. Schematic diagram for multi-layer grids.

There are three layers of grids in Figure 4. The light blue frame represents the first layer, the brown frame indicates the second layer and the dark blue frame corresponds to the third layer. The two black circles denote the solid boundaries within the flow field. The first layer grid is half the size of the second, and this pattern continues.

To facilitate grid division and parallel computation, this paper adopts a center-point format for storing grid point information. The resulting storage format is similar to a fork-tree structure [12], which facilitates programming to process.

In a locally refined multi-layer grid structure, the ratio of the coarse grid size to the fine grid size is set to $n=\delta {\mathit{r}}_c/\delta {\mathit{r}}_s=2$, where c represents a coarse grid and s represents a fine grid. Thus, in a 2D problem, a coarse grid can be subdivided into four fine grid cells. To ensure fluid viscosity consistency throughout the flow field, the following relationships need to be satisfied:

$${\delta }_t^s={\displaystyle \frac{1}{n}}{\delta }_t^c,{\tau }^s={\displaystyle \frac{1}{2}}+n\left({\tau }^c-{\displaystyle \frac{1}{2}}\right)$$

(16)

Velocity continuity needs to be satisfied at the intersection interface of grids of different sizes. Therefore, it must be satisfied that the equilibrium state distribution equations have the same value on the grid intersections. The following relationship is obtained:

$$f_{i,\mathrm{c}}^{\mathrm{neq}}=n{\displaystyle \frac{{\tau }_{\mathrm{c}}}{{\tau }_s}}{f}_{i,s}^{\mathrm{neq}}$$

(17)

A local grid refinement method based on distribution functions and bi-directional coupling was developed by Dupuis and Chopard [23]. In combination with the evolution of the RLBM, the paper establishes the relationship between the fine grid and the coarse grid.

$$f_{i,c}^{+}=f_{i,s}^{\mathrm{eq}}+n{\displaystyle \frac{{\tau }_c-1}{{\tau }_s-1}}(f_{i,s}-f_{i,s}^{\mathrm{eq}})$$

(18)

$$f_{i,s}^{+}=f_{i,\mathrm{c}}^{\mathrm{eq}}+{\displaystyle \frac{{\tau }_s-1}{n({\tau }_{\mathrm{c}}-1)}}(f_{i,\mathrm{c}}-f_{i,\mathrm{c}}^{\mathrm{eq}})$$

(19)

where $f_{i,c}^{+}$ is obtained by interpolating [40] the coarse grid through the fine grid and $f_{i,s}^{+}$ is obtained by interpolating the fine grid through the coarse grid. To ensure the moments of the coarse and fine grids are identical, the coarse grids perform one collision and streaming, while fine grids require two collisions and streamings.

The interpolation process between grids of different scales is shown in Figure 5. Firstly the grid information needs to be initialized and a pre-collision step is performed. Subsequently, the coarse grid undergoes one collision and streaming, while the fine grid undergoes two collisions and streamings. An interpolation between fine and coarse grids is conducted. This process is repeated until termination.

Figure 5. Schematic diagram of the interpolation process.

3. Parallel Strategy and Algorithm

This paper describes the division of the grid in the two-dimensional case. A load balancing strategy [34] prevents a process from going into an idle state while waiting for data from other processes. When performing simulations, all grid cells are assigned to available processes. The presence of idle processes is prevented. The effective assignment of grids to processes is essential for efficiency improvements.

In the simulation scenario, the grid for each layer is stored in a one-dimensional array. This array stores the number of grids involved in each column of the evolution step for the grid region. The one-dimensional array is partitioned according to the number of processes.

Firstly, this paper separates the different grid layers into individual dimensions. Divide the various layers of the grid into multiple subdomains along the same direction. Subdomains with the same number of processes in different layers are formed into a separate computational region. At the same time, data exchange between boundary grids basically occurs only between neighboring processes. In this paper, the grid division algorithm is designed in Algorithm 1. The flowchart for grid division is shown in Figure 6.

Algorithm 1 Load Balancing-Based Grid Dividing Algorithm

Input: Number of grid layers: N; Number of processes: $np$; Output: File after grid division.  1: int $Col\left[N\right]\left[R\right]=0$, $Total\left[N\right]\left[R\right]=0$;  2: // R is the number of columns in the current layer grid.  3: //$Col\left[N\right]\left[R\right]$ represents how many grids are stored in each of the columns 1 to R from layer 1 to layer N.  4: //$Total\left[N\right]\left[R\right]$ represents how many grids are stored in column 1 to the current column.  5: for int $k=0$ to $N-1$ do  6:     for int $j=0$ to $R-1$ do  7:         $Col\left[k\right]\left[j\right]=l$; //l is the number of grids in the $jth$ column of the $kth$ layer.  8:     end for  9: end for 10: for int $k=0$ to $N-1$ do 11:     for int $j=0$ to $R-1$ do 12:         $Total\left[k\right]\left[j\right]=Total\left[k\right][j-1]+Col\left[k\right]\left[j\right]$; 13:     end for 14: end for 15: for int $id=0$ to $np-1$ do 16:     for int $k=0$ to $N-1$ do 17:         for int $j=0$ to $R-1$ do 18:            if $Total\left[k\right]\left[j\right]>Total\left[k\right][R-1]\ast (id+1)/np$ then 19:                int $a=Total\left[k\right]\left[j\right]-Total\left[k\right][R-1]\ast (id+1)/np$; 20:                int $b=Total\left[k\right][R-1]\ast (id+1)/np-Total\left[k\right][j-1]$; 21:                if a < b then 22:                    Set the process number in this column to $id$; 23:                    ++$id$; 24:                else 25:                    ++$id$; 26:                    Set the process number in this column to $id$; 27:                end if 28:            else 29:                Set the process number in this column to $id$; 30:            end if 31:         end for 32:     end for 33: end for

Figure 6. Flow chart for grid division.

The division of the multi-layer grid is shown schematically in Figure 7.

Figure 7. Schematic diagram of the multi-layer grid division.

The three-layer grid structure of a two-dimensional flow over two circular cylinders is taken as an example in this paper. Using the multi-layer grids RLBM, all grid points within the same layer have similar computational demands. Coarse grid points undergo half the computations compared to fine grid points. The grid area in Figure 7 consists of three distinct grid sizes. Based on the division idea in Algorithm 1, a parallel grid division was performed. Grids of varying resolutions are partitioned into four equal segments. Each MPI process manages a segment of each grid layer. The data transfer method utilizing this parallel partitioning is illustrated in Figure 7. Red denotes process 0, yellow denotes process 1, purple denotes process 2, and blue denotes process 3.

The data-transfer method of the multi-layer grid is shown schematically in Figure 8.

Figure 8. Schematic diagram of MPI data transfer method.

Under this parallel strategy, each process exchanges data solely with two neighboring processes. The parallel algorithm proposed in this paper relies on the grid division algorithm within the computational domain. After dividing the grid into sections, the data at the grid junctions include data for evolution and data for interpolation. The data used for evolution are from the same layer of grid connections, the outermost vertical space on the grid. The data used for interpolation are from the intersection of different layers of the grid. These data are used to complete the interpolation between coarse and fine grids. Both the data used for transfer and the data used for interpolation contain information about the neighboring grids and the values of the distribution function. Parallel processing can be carried out after data transfer of interpolated and transmitted data. In this section, Algorithm 2 gives an MPI-based parallel algorithm for RLBM on multi-layer grids.

To prevent deadlocks from occurring, each grid performs separate collisions, streaming, and computation of macro-physical quantities within the computational domain for each process. Neighboring processes pass the information on the boundary grid at the grid junction and then compute. This optimizes communication efficiency. Each process first performs a local computation. Finally, the results of the computations in each region are restored to the results of the complete original problem.

Algorithm 2 MPI-based parallel algorithm for RLBM on multi-layer grids

Input: Grid files after division, total number of MPI processes $np$, number of grid layers N, Initial information of the flow field. Output: Evolution results.      1. Calculate the number of grid points in each layer of the multi-layer grids.      2. The multi-layer grids are uniformly divided into $np$ copies of grid regions according to Algorithm 1.      3. Allocate a grid area to each process in the MPI.      4. From layer N to layer 1, each process executes the RLBM based on multi-layer grid.      5. Once the grid calculations are complete at each layer, a synchronisation operation is performed.      6. Determine whether the exit condition has been met, if not, continue with step 4 and step 5.      7. Output calculation results.

4. Numerical Simulations

This section provides two types of numerical validation: flow over two circular cylinders in the 2D case and flow around a sphere in the 3D case. The parallel performance of 3D numerical experiments is evaluated and analyzed.

4.1. Flow over Two Circular Cylinders in a 2D Case

To demonstrate the precision of the parallel algorithm of RLBM based on multi-layer grids, the flow over two circular cylinders at different spacing ratios was simulated. Two cylinders of the same size are placed in the channel. The diameter of the cylinder is defined as D. The Reynolds number was set the same in all scenarios of this experiment at Re = 200. The length of the channel is 80D and the width of the channel is 60D. The channel’s inlet velocity is U (U = 0.1). The front cylinder is located 20D from the left boundary. The back cylinder is positioned at a distance L from the front cylinder. This experiment investigates the impact on the flow field as the position of the rear cylinder changes. The distance L between two cylinders is taken as 1.5D, 2D, 3D, and 4D. A schematic of the simulation scenario is shown in Figure 9.

Figure 9. Schematic diagram of two-dimensional flow over two circular cylinders.

The fluid region grid is set up as follows: The size of the refined two-layer grid area is $50D\times 35D.$ and $20D\times 15D.$, respectively. Each layer’s grid size is half that of the preceding layer. Boundary grid refinement was performed around the two cylinders to capture flow details. The three-layer grid structure is shown in Figure 10.

Figure 10. Three-layer grid structure diagram.

The flow field is analyzed for different spacing ratios as follows: As the evolutionary steps increase, the vortices between the cylinders gradually stabilize. The streamlines of the upstream cylinders closely follow the downstream cylinders. A symmetrical vortex pair was created between the two cylinders. The vortex of the downstream cylinder could not form a stable vortex pair due to the interference of the upstream cylinder, as shown in Figure 11, Figure 12, Figure 13 and Figure 14. The different colors of velocity contours represent different ranges of velocity values.

Figure 11. Velocitycontours and streamlines diagram at $L/D=1.5$.

Figure 12. Velocity contours and streamlines diagram at $L/D=2$.

Figure 13. Velocity contours and streamlines diagram at $L/D=3$.

Figure 14. Velocity contours and streamlines diagram at $L/D=4$.

The vortex between the two cylinders cannot be stable and symmetrical when the spacing ratio (L/D = 3). The existence of critical spacing is proved [41].

In this paper, the lift coefficients $C_L$ and the drag coefficients $C_D$ are used to verify the accuracy of the algorithm when performing numerical calculations.

$$C_D={\displaystyle \frac{F_D}{\frac{1}{2}\rho U^{2}D}}$$

(20)

$$C_L={\displaystyle \frac{F_L}{\frac{1}{2}\rho U^{2}D}}$$

(21)

Table 1 and Table 2 demonstrate the results obtained using the algorithm of this paper in comparison with the published literature [41,42,43].

Table 1. $C_L$ and $\overline{C_D}$ of the front cylinder when Re = 200.

	Koda et al. [42]		Alam et al. [41]		Meneghini et al. [43]		Present Work
$\mathit{L}\mathsf{/}\mathit{D}$	$\overline{{\mathit{C}}_{\mathit{D}\mathbf{1}}}$	${\mathit{C}}_{\mathit{L}\mathbf{1}}$	$\overline{{\mathit{C}}_{\mathit{D}\mathbf{1}}}$	${\mathit{C}}_{\mathit{L}\mathbf{1}}$	$\overline{{\mathit{C}}_{\mathit{D}\mathbf{1}}}$	${\mathit{C}}_{\mathit{L}\mathbf{1}}$	$\overline{{\mathit{C}}_{\mathit{D}\mathbf{1}}}$	${\mathit{C}}_{\mathit{L}\mathbf{1}}$
1.5	1.107	0.016	−	−	1.06	−	1.070	0.020
2	−	−	−	−	1.03	−	1.052	0.047
3	1.043	0.023	1.018	0.022	1.00	−	1.043	0.022
4	1.267	0.532	1.255	0.551	1.18	−	1.256	0.547

Table 2. $C_L$ and $\overline{C_D}$ of the rear cylinder when Re = 200.

	Koda et al. [42]		Alam et al. [41]		Meneghini et al. [43]		Present Work
$\mathit{L}\mathsf{/}\mathit{D}$	$\overline{{\mathit{C}}_{\mathit{D}\mathbf{2}}}$	${\mathit{C}}_{\mathit{L}\mathbf{2}}$	$\overline{{\mathit{C}}_{\mathit{D}\mathbf{2}}}$	${\mathit{C}}_{\mathit{L}\mathbf{2}}$	$\overline{{\mathit{C}}_{\mathit{D}\mathbf{2}}}$	${\mathit{C}}_{\mathit{L}\mathbf{2}}$	$\overline{{\mathit{C}}_{\mathit{D}\mathbf{2}}}$	${\mathit{C}}_{\mathit{L}\mathbf{2}}$
1.5	−0.201	0.041	−	−	−0.08	−	−0.211	0.056
2	−	−	−	−	−0.17	−	−0.231	0.189
3	−0.130	0.209	−0.130	0.212	−0.08	−	−0.132	0.267
4	0.482	1.210	0.360	1.117	0.380	−	0.380	1.117

Figure 15, Figure 16, Figure 17 and Figure 18 represent different spacings of the two cylinders. The cyclic variation of the lift coefficients $C_L$ and drag coefficients $C_D$ of the fluid flowing through the cylinders with time iteration steps.

Figure 15. Cylindrical bypass: $C_D$ and $C_L$ when Re = 200 and $L/D=1.5$. (a) Front cylinder; (b) rear cylinder.

Figure 16. Cylindrical bypass: $C_D$ and $C_L$ when Re = 200 and $L/D=2$. (a) Front cylinder; (b) rear cylinder.

Figure 17. Cylindrical bypass: $C_D$ and $C_L$ when Re = 200 and $L/D=3$. (a) Front cylinder; (b) rear cylinder.

Figure 18. Cylindrical bypass: $C_D$ and $C_L$ when Re = 200 and $L/D=4$. (a) Front cylinder; (b) rear cylinder.

In Figure 19, the number of grids and CPU runtime are compared for a single-layer grid and a three-layer grid. The CPU time for a three-layer grid is only 1/8th of the original, while the number of grids to be computed is 1/4th of the original. The results demonstrate that the multi-layer grids regularized lattice Boltzmann method (RLBM) exhibits high accuracy and stability.

Figure 19. Comparison of the running overhead of single-layer grid and multi-layer grid.

4.2. Flow around a Sphere in 3D Case

To demonstrate the precision of the parallel algorithm of RLBM based on multi-layer grids, flow around a sphere was simulated. To place a sphere in a closed box. The radius of the sphere is defined as R. The box size is $16R\times 16R\times 25R$. The inlet flow velocity is U(U = 0.1). The setup of the simulation scene is shown in Figure 20. A three-layer grid was used to perform the numerical calculations. The dimension of the first layer of the grid is 1/6R. The size of the second layer of the grid region is $10R\times 10R\times 15R$. The size of the third layer of the grid area is $6R\times 6R\times 10R$.

Figure 20. Velocity contours and streamlines diagram at $L/D=1.5$.

In this section, experiments were conducted with four sets of Reynolds numbers of 50, 100, 150, and 200. The length $X_S$ of the recirculation region is different under different conditions. The flow diagrams at different Reynolds numbers are shown in Figure 21, Figure 22, Figure 23 and Figure 24. The length $X_S$ of the recirculation region under different conditions is compared with the published literature [44,45,46] in Table 3.

Figure 21. Streamlines of the flow field of a three-dimensional sphere at Re = 50.

Figure 22. Streamlines of the flow field of a three-dimensional sphere at Re = 100.

Figure 23. Streamlines of the flow field of a three-dimensional sphere at Re = 150.

Figure 24. Streamlines of the flow field of a three-dimensional sphere at Re = 200.

Table 3. Drag coefficients $C_D$ and the length $X_S$ of the recirculation region.

	Cheng et al. [45]		Wu et al. [46]		Xu et al. [44]		Present Work
$\mathit{Re}$	$\overline{{\mathit{C}}_{\mathit{D}}}$	${\mathit{X}}_{\mathit{S}}$	$\overline{{\mathit{C}}_{\mathit{D}}}$	${\mathit{X}}_{\mathit{S}}$	$\overline{{\mathit{C}}_{\mathit{D}}}$	${\mathit{X}}_{\mathit{S}}$	$\overline{{\mathit{C}}_{\mathit{D}}}$	${\mathit{X}}_{\mathit{S}}$
50	−	−	−	−	1.56	0.43	1.575	0.44
100	1.099	−	1.128	−	1.12	0.90	1.08	0.925
150	−	−	−	−	0.85	1.26	0.881	1.25
200	0.800	−	0.800	−	0.81	1.42	0.783	1.40

In this paper, the drag coefficients $C_D$ and length $X_S$ of the recirculation region are used to verify the accuracy of the algorithm when performing numerical calculations.

$$C_D={\displaystyle \frac{8F_x}{\rho \pi U^2{D}^2}}$$

(22)

Table 3 demonstrates the results obtained using the algorithm of this paper in comparison with the published literature [44,45,46].

4.3. Performance Evaluation

In this section, we evaluate the algorithm proposed in this paper in terms of both speed-up ratio and efficiency. The scenario configuration for the simulation in this section is the same as in Section 4.2. This section evaluates parallel performance using speedup ratio and efficiency. The speedup ratio is defined as follows:

$$Speedup={\displaystyle \frac{T_{serial}}{T_{parallel}\left(p\right)}}$$

(23)

where $T_{serial}$ is the serial execution time and $T_{parallel}\left(p\right)$ is the parallel execution time using p cores.

The efficiency is defined as follows:

$$Efficiency={\displaystyle \frac{Speedup}{p}}$$

(24)

The speedup ratio and efficiency of the parallel algorithm proposed in this paper at different numbers of cores is demonstrated in Table 4.

Table 4. The speedup ratio and efficiency of the algorithm proposed in this paper.

Cores	1	2	4	8	16
Speedup	1	1.73	3.37	5.77	9.05
Efficiency	100%	86.51%	84.25%	72.13%	56.56%

In order to facilitate the comparison of the performance of the algorithms in this paper, parallel computation is performed using OpenMP under the same experimental conditions and scenarios. The speedup ratio and efficiency of the OpenMP at different numbers of cores is demonstrated in Table 5.

Table 5. The speedup ratio and efficiency of the OpenMP.

Cores	1	2	4	8	16
Speedup	1	1.46	2.35	3.05	4.62
Efficiency	100%	73.00%	58.75%	38.18%	28.88%

The performance of the multi-layer grids RLBM algorithm using the parallel algorithm proposed in this paper is compared with the multi-layer grids RLBM algorithm using OpenMP multi-threaded computation. The comparison of speed-up ratios is shown schematically in Figure 25. The comparison of efficiency is shown schematically in Figure 26.

Figure 25. Comparison of the speed-up between the parallel algorithm in this paper and the parallel algorithm using OpenMP.

Figure 26. Comparison of the efficiency between the parallel algorithm in this paper and the parallel algorithm using OpenMP.

It is clear that the parallel algorithm proposed in this paper exhibits a significant speed-up ratio for all numbers of cores. For example, in the case of 16 cores, the algorithm achieves a speed-up ratio of 9.05, while the speed-up ratio using OpenMP is only 4.621. It shows that the algorithm proposed in this paper is able to make better use of parallel computing resources and achieve higher computational efficiency when scaling up to more processor cores. As the number of processor cores increases, the speed-up ratio of MPI increases significantly more than that of OpenMP, demonstrating excellent scalability.

The algorithm proposed in this paper is significantly more efficient than OpenMP for all cores. In the two-core case, the MPI is 87% efficient compared to 73% for the OpenMP. This trend is more pronounced as the number of cores increases. It shows that the algorithm in this paper is better able to maintain efficient load balancing and low communication overhead when increasing the number of cores. Although the efficiency of the MPI algorithm decreases as the number of cores increases, the decrease is significantly smaller than that of the OpenMP algorithm. The efficiency of the MPI algorithm is still maintained at a reasonable level in larger-scale parallel computations, indicating the effectiveness of its load-balancing strategy.

The rationality and efficiency of the algorithm in this paper are proved, and it can greatly reduce the computational bottleneck of individual processes and improve the overall computational efficiency. The algorithm only needs to communicate between neighboring processes and only needs to exchange information about the boundary grid. This mode of communication greatly reduces the amount of communication and allows the parallel algorithm to remain highly efficient at larger scales.

5. Conclusions

The regularized lattice Boltzmann method is an improvement based on the lattice Boltzmann method. It can improve accuracy without significantly increasing computational overheads. The multi-layer grids method can be effectively combined with RLBM. Focusing the fine grid on the parts that require more accurate calculations. The aim is to improve the computational accuracy and stability. A large number of grids is usually required for numerical calculations. RLBM has a natural parallelism; therefore, designing a parallel algorithm is necessary. In this paper, a load-balancing grid division algorithm is designed. It can achieve computational load balancing among the processes and minimize the communication overhead. Based on the above algorithm, an MPI-based parallel algorithm for RLBM on multi-layer grids is proposed. It can obtain the numerical results in a faster way. The accuracy and feasibility of the proposed method are verified by numerical experiments on flow over two circular cylinders in the 2D case and flow around a sphere in the 3D case. The experimental results show that the computation time is significantly reduced as the number of processes increases. Parallel algorithms have excellent performance in terms of speedup. In this paper, the proposed algorithm is compared with the use of OpenMP in a 3D flow around a sphere simulation scenario. The speed-up is close to 60 percent when using 16 cores. Using MPI is 28% more efficient than the parallel method using OpenMP. The algorithm performs well in large-scale computing tasks. Meanwhile, the load balancing strategy effectively reduces communication overhead, maintaining high parallel efficiency. In conclusion, the parallel algorithm on a regularized lattice Boltzmann method for multi-layer grids proposed in this paper offers an effective solution for massively parallel computing. The accuracy and stability of the algorithm will be tested in more CFD scenarios in the future, such as multiple cylinder arrangement, moving boundary cases, etc.