A Code for Simulating Heat Transfer in Turbulent Channel Flow

Abstract

One numerical method was designed to solve the time-dependent, three-dimensional, incompressible Navier–Stokes equations in turbulent thermal channel flows. Its originality lies in the use of several well-known methods to discretize the problem and its parallel nature. Vorticy-Laplacian of velocity formulation has been used, so pressure has been removed from the system. Heat is modeled as a passive scalar. Any other quantity modeled as passive scalar can be very easily studied, including several of them at the same time. These methods have been successfully used for extensive direct numerical simulations of passive thermal flow for several boundary conditions.

1. Introduction

Turbulence is probably the open problem in physics with most applications in daily life. Turbulent flows are intrinsic to almost any flow in engineering, but they are also extremely important in meteorology or the dispersion of contaminants. It is well known that we still even lack an existence and uniqueness theorem about the solution of the governing equations of turbulent flows, the Navier–Stokes equations. Moreover, these equations cannot be solved analytically apart from some trivial examples. It is generally admitted that the simulation can be considered at three levels of detail, Reynolds Average Navier Stokes (RANS), Large Eddy Simulation (LES), and Direct Numerical Simulations (DNS) [1] which also corresponds to different levels of precision. Both RANS and LES models can simulate typical industrial flows with the necessary accuracy and in practical computation times [2,3,4,5], but DNS is until now the only trustworthy method to investigate physical properties of flows which are little understood.

However, DNS are extremely expensive as every scale of turbulence, both temporal and spatial ones, has to be properly resolved. This implies very fine grids and small temporal steps. Moreover, and as a consequence of Kolmogorov’s scales, [6,7], the number of points needed to carry out the simulation grows extremely fast, as $R{e}^{9/4}$, where $Re$ is the Reynolds number. Thus, to use DNS to understand turbulent flows, very simple geometries are employed, allowing very precise and fast numerical tools that will be described in this paper. Canonical wall-bounded flows are pipes, channel flows, and the flat-plate boundary layer. These flows are in essence theoretical abstractions and do not appear as such in reality. However, they constitute the basic blocks of more complete real flows and have been studied for 80 years. For isothermal flows, experiments of these canonical flows have been instrumental in the development of the theory since the birth of boundary layer research. In particular, they remain important for the study of very high Reynolds numbers, as DNS is still not able to reach Reynolds numbers as high as those of experiments. High Reynolds numbers are needed to obtain a clear separation of scales related to the near-wall turbulence cycle. The main control parameter is the friction Reynolds Number $R{e}_{\tau }=h{u}_{\tau }/\nu$, where $\nu$ is the viscosity, h is a characteristic length and $u_{\tau }$ is the friction velocity at the wall. The largest experiment ever done has been published in 2018 for a friction Reynolds number of 2e4 [8], a value that is absolutely out of reach for DNS. That said, simulations of wall-bounded turbulence are particularly helpful in identifying physical processes occurring in near-wall turbulence as the whole velocity field (and its derivatives) is available. Moreover, experiments showing the 3D thermal field of a turbulent flow are almost impossible to do. For thermal flows, DNS can give insight where experimental flows are impossible.

The first wall-bounded simulation was run by Kim, Moin, and Moser in 1987 [9] for channel flows and in 1988 by Spalart [10] for Turbulent Boundary Layers. Since then, the friction Reynolds number simulated has been continuously growing [11,12,13,14,15,16,17,18,19]. Some of these simulations, Refs. [12,16,18,19] were made with an earlier version of this code. The history is similar for thermal passive flows, but the Reynolds number reached grew up at a slower pace. The first DNS of a thermal flow was carried out by Kim and Moin in 1987 [20]. Kim and Moin obtained first-order turbulence statistics for $R{e}_{\tau }=180$ and several Prandtl numbers, $Pr=0.1,$ 1 and 2. The Prandtl number is the ratio between momentum diffusivity to thermal diffusivity. They also computed a very important number for modeling, the turbulent Prandtl number. In addition, for the Prandtl number of air, $Pr=0.71$, correlations between the velocity and the temperature were also calculated. A somewhat artificial boundary condition was imposed in which heat was generated internally and removed from both cold isothermal walls. This condition for the thermal field plays an analogous role to that of the pressure gradient does for the velocity field. Later, Lyons et al. [21] performed a simulation for $R{e}_{\tau }=150$ and $Pr=1$. The boundary condition used in this later work consisted of both walls kept at different temperatures. Finally, Kasagi et al. [22] performed a DNS for $R{e}_{\tau }=150$ and $Pr=0.71$ with a more realistic boundary condition, the Mixed Boundary Condition (MBC from now on). For this condition, the average heat flux over both heating walls is constant and the temperature increases linearly in the streamwise direction. The instantaneous heat flux may vary with respect to time and position. In the work done by Piller [23], it was shown that the MBC acts as an ideal isothermal boundary condition in the inner layer and as an ideal isoflux boundary condition (fixed heat flux) in the outer layer.

After these simulations were made, the trend has been to increase the friction Reynolds number for different molecular Prandtl numbers. However, values of $R{e}_{\tau }$ and $Pr$ are limited by the computational cost. Yano and Kasagi [24] stated that the computational cost can be approximated as $L_x^2{L}_{y}R{e}_{\tau }^{4}P{r}^{3/2}$. Moreover, even if only one equation has to be added to the system, this equation is also nonlinear. Typically, the cost of accurately computing the nonlinear term accounts for 80 to $90\%$ of the total time of the simulation. Thus, to add the energy equation we need to double the computational time, and the Reynolds number achieved is still low for the majority of Prandtl numbers.

In this paper, we will restrict ourselves to explain the numerical code and its implementation. Since 2018, several works have been published using this algorithm, which was written in Fortran90. In Lluesma et al. [25], the authors validated the code and found the optimal size of the computational box for Reynolds numbers up to $R{e}_{\tau }=2000.$ The range of Prandtl numbers, below $Pr=0.7,$ was increased in Alcántara-Ávila [26]. Later, in [27], the large structures of Couette flow were analyzed. In [28,29], the authors were able to increase the study for $Pr>0.7,$ reaching $Pr=10,$ and $R{e}_{\tau }=5000$ for $Pr=0.7.$ Finally, this code has been used to study the large structures found in stratification problems, cite [30].

The structure of the paper is as follows. In the next section, the numerical problem is described. The third section is devoted to explain the numerical techniques used for discretization both in time and space. The fourth section describes the parallelization techniques used. Finally, conclusions and future works are outlined.

2. Methods

The flow considered in this work is a turbulent channel flow driven by a pressure gradient (Poiseuille flow). The flow is treated as incompressible. The thermal field is treated as a passive scalar. As mentioned before, the boundary condition used for the thermal field is the MBC. For this boundary condition, an uniform heat flux is heating both walls, which introduces the heat into the flow. The temperature of these walls increases linearly in the streamwise direction and does not depend on time.

In Figure 1, a schematic representation of the lower half of the computational box used can be observed. In this plot, contours of an instantaneous snapshot of the streamwise velocity are represented, colored by the magnitude of the velocity. The flow moves from left to right. Periodic conditions are imposed in the streamwise and spanwise boundaries.

As it was said above, the computational box was optimized in [25]. The dimensions of the box are $L_x=2\pi h$, $L_y=2h,$ and $L_z=\pi h$ in the streamwise, wall-normal, and spanwise directions, respectively. Coordinates in these directions are denoted by x, $y,$ and z. The corresponding velocities are U, $V,$ and W, or, using index notation, $U_i$. This is also the case for vorticity, $\overrightarrow{\mathsf{\Omega }}=\overrightarrow{\nabla }\times \overrightarrow{U}=\left({\mathsf{\Omega }}_1,{\mathsf{\Omega }}_2,{\mathsf{\Omega }}_3\right)=\left({\mathsf{\Omega }}_x,{\mathsf{\Omega }}_y,{\mathsf{\Omega }}_z\right)$ and helicity $\overrightarrow{H}=\overrightarrow{U}\times \overrightarrow{\mathsf{\Omega }}=\left(H_1,H_2,H_3\right)=\left(H_x,H_y,H_z\right)$. Temperature is represented by T. However, the transformed temperature, $\mathsf{\Theta }$ (defined below), will be used during the entire paper. Uppercase letters denote instantaneous flow magnitudes. Using the Reynolds decomposition, one can obtain the averaged value, denoted by an overbar, and the fluctuating part, denoted by a lowercase letter, of the flow magnitudes, i.e., $U=\overline{U}+u$. The superscript + indicates normalization in wall units, using $\nu$ and $u_{\tau }=\sqrt{{\tau }_w/\rho }$, where ${\tau }_w$ is the mean shear stress and $\rho$ is the fluid density.

The behavior of turbulent flows is described by the Navier–Stokes equations, which are composed by the continuity and momentum equations,

$$\begin{array}{ccc}\hfill {\partial }_j{U}_j^{+}& =& 0,\hfill \end{array}$$

(1)

$$\begin{array}{ccc}\hfill {\partial }_t{U}_i^{+}+U_j^{+}{\partial }_j{U}_i^{+}& =& {\displaystyle -{\partial }_i{P}^{+}+\frac{1}{R{e}_{\tau }}{\partial }_{jj}{U}_i^{+},}\hfill \end{array}$$

(2)

and the energy equation,

$${\partial }_t{\mathsf{\Theta }}^{+}+U_j^{+}{\partial }_j{\mathsf{\Theta }}^{+}=\frac{1}{R{e}_{\tau }Pr}{\partial }_{jj}{\mathsf{\Theta }}^{+}+U^{+}/{\langle U^{+}\rangle }_{xyz},$$

(3)

where ${\langle U^{+}\rangle }_{xyz}$ is the average of $U^{+}$ in time and in the three spatial directions, i.e., ${\langle U^{+}\rangle }_{xyz}=U_b^{+}$, where $U_b^{+}$ is the bulk velocity. In the energy equation, the transformed temperature, $\mathsf{\Theta }=T_w-T$ is used. Since the MBC is used, the temperature in the channel increases linearly in the streamwise direction and periodic conditions cannot be used for T. $T_w$ contains the nonperiodic part of the temperature, which makes $\mathsf{\Theta }$ periodic in the streamwise direction. Notice that the method described below is also useful for any other passive scalar, as it could be NOx concentration. It is also important to point out that for clarity, only the system with one energy equation will be explained here. However, as heat transfer is treated as a passive scalar, several different Prandtl numbers can be run simultaneously. In what follows and to facilitate the discussion, + superscripts will be omitted.

Using some algebra, which involves taking the rotational of Equations (1) and (2) twice, these equations can be written in velocity-vorticity form obtaining a fourth-order equation for V

$${\partial }_t{\nabla }^{2}V=-h_v+\frac{1}{R{e}_{\tau }}{\nabla }^{4}V,$$

(4)

and a second-order equation for ${\mathsf{\Omega }}_y$

$${\partial }_t{\mathsf{\Omega }}_y=-h_g+\frac{1}{R{e}_{\tau }}{\nabla }^2{\mathsf{\Omega }}_y,$$

(5)

where $h_v$ and $h_g$ collect the nonlinear part of the Equations (4) and (5), and are given by

$$\begin{array}{ccc}\hfill h_v& =& -{\partial }_y\left({\partial }_x{H}_1+{\partial }_z{H}_3\right)+\left({\partial }_{xx}+{\partial }_{zz}\right)H_2,\hfill \\ \hfill h_g& =& {\partial }_z{H}_1-{\partial }_x{H}_3,\hfill \end{array}$$

The main advantage of this system is that pressure is not present, simplifying the problem. Notice that one can easily recover velocities and vorticities from the continuity equation and the definition of ${\mathsf{\Omega }}_y$,

$$\begin{array}{ccc}\hfill {\partial }_{x}U+{\partial }_{z}W& =& -{\partial }_{y}V,\hfill \end{array}$$

(6)

$$\begin{array}{ccc}\hfill {\partial }_{z}U-{\partial }_{x}W& =& {\mathsf{\Omega }}_y,\hfill \end{array}$$

(7)

Taking derivatives,

$$\begin{array}{ccc}\hfill {\partial }_x\left({\partial }_{x}U+{\partial }_{z}W\right)+{\partial }_z\left({\partial }_{z}U-{\partial }_{x}W\right)& =& {\partial }_x\left(-{\partial }_{y}V\right)+{\partial }_z\left({\mathsf{\Omega }}_y\right),\hfill \end{array}$$

(8)

$$\begin{array}{ccc}\hfill {\partial }_z\left({\partial }_{x}U+{\partial }_{z}W\right)-{\partial }_x\left({\partial }_{z}U-{\partial }_{x}W\right)& =& {\partial }_z\left(-{\partial }_{y}V\right)+{\partial }_x\left({\mathsf{\Omega }}_y\right),\hfill \end{array}$$

(9)

and reorganizing the terms, we get

$$\begin{array}{ccc}\hfill ({\partial }_{xx}+{\partial }_{zz})U& =& -{\partial }_{yx}V+{\partial }_z{\mathsf{\Omega }}_y,\hfill \end{array}$$

(10)

$$\begin{array}{ccc}\hfill ({\partial }_{xx}+{\partial }_{zz})W& =& -{\partial }_{yz}V-{\partial }_x{\mathsf{\Omega }}_y.\hfill \end{array}$$

(11)

As the flow is periodic in both x and $z,$ it is natural to use Fourier methods in these directions. The Fourier transform of any field $\phi \left(x,y,z\right)$ is defined as

$$\phi \left(x,y,z\right)=\sum _{k_x}\sum _{k_z}{\widehat{\phi }}_{k_x{k}_z}\left(y\right)e^{i\left(k_{x}x+k_{z}z\right)},$$

where the hat indicates the Fourier coefficient, and $k_x$ and $k_z$ are the wavenumbers in x and $z,$ respectively. Equations (10) and (11) can be trivially solved as they become

$$\begin{array}{ccc}\hfill {\widehat{U}}_{k_x{k}_z}& =& \frac{-i{k}_x{\partial }_y{\widehat{V}}_{k_x{k}_z}+i_{k_z}{\widehat{\mathsf{\Omega }}}_y}{-(k_x^2+k_z^2)},\hfill \end{array}$$

(12)

$$\begin{array}{ccc}\hfill {\widehat{W}}_{k_x{k}_z}& =& \frac{-i_{k_z}{\partial }_y{\widehat{V}}_{k_x{k}_z}-i{k}_x{\widehat{\mathsf{\Omega }}}_{y_{k_x{k}_z}}}{-(k_x^2+k_z^2)}.\hfill \end{array}$$

(13)

Notice that this technique cannot be used for the $\left(0,0\right)$-modes of U and $W.$ Using the Navier–Stokes equations, one can write

$$\begin{array}{ccc}\hfill {\partial }_t{\widehat{U}}_{00}& =& -{\partial }_x\widehat{P}+{\widehat{H}}_{1,00}+\frac{1}{R{e}_{\tau }}{\partial }_{yy}{\widehat{U}}_{00},\hfill \\ \hfill {\partial }_t{\widehat{W}}_{00}& =& -{\partial }_z\widehat{P}+{\widehat{H}}_{3,00}+\frac{1}{R{e}_{\tau }}{\partial }_{yy}{\widehat{W}}_{00},\hfill \end{array}$$

and solve these equations instead. The pressure gradient in the spanwise direction is negligible. However, the pressure gradient in the streamwise component can be used to keep the flow mass constant. Using Fourier transforms, Equations (3)–(5) are transformed into Fourier space in x and z, obtaining three decoupled problems,

$$\begin{array}{ccc}\hfill {\partial }_t{\widehat{\varphi }}_{k_x{k}_z}& =& {\widehat{h}}_{v,k_x{k}_z}+\frac{1}{R{e}_{\tau }}{\nabla }^2{\widehat{\varphi }}_{k_x{k}_z},\hfill \end{array}$$

(14)

$$\begin{array}{ccc}\hfill {\nabla }^2{\widehat{V}}_{k_x{k}_z}& =& {\widehat{\varphi }}_{k_x{k}_z},\hfill \end{array}$$

(15)

$$\begin{array}{ccc}\hfill {\widehat{V}}_{k_x{k}_z}\left(\pm 1\right)& =& 0,{\partial }_y{\widehat{V}}_{k_x{k}_z}\left(\pm 1\right)=0,\hfill \end{array}$$

(16)

$$\begin{array}{ccc}\hfill {\partial }_t{\widehat{\mathsf{\Omega }}}_{y,k_x{k}_z}& =& {\widehat{h}}_{g,k_x{k}_z}+\frac{1}{R{e}_{\tau }}{\nabla }_{}^2{\widehat{\mathsf{\Omega }}}_{y,k_x{k}_z},\hfill \end{array}$$

(17)

$$\begin{array}{ccc}\hfill {\widehat{\mathsf{\Omega }}}_{y{k}_x{k}_z}(\pm 1)& =& 0,\hfill \end{array}$$

(18)

$$\begin{array}{ccc}\hfill {\partial }_t{\widehat{\mathsf{\Theta }}}_{k_x{k}_z}& =& {\widehat{h}}_{t,k_x{k}_z}+\frac{1}{R{e}_{\tau }Pr}{\nabla }^2{\widehat{\mathsf{\Theta }}}_{k_x{k}_z},\hfill \end{array}$$

(19)

$$\begin{array}{ccc}\hfill {\widehat{\mathsf{\Theta }}}_{k_x{k}_z}(\pm 1)& =& 0,\hfill \end{array}$$

(20)

where ${\widehat{h}}_{t,k_x{k}_z}$ is the Fourier transform of the energy equation nonlinear term.

Problem (14)–(16) is usually solved splitting ${\widehat{V}}_{k_x{k}_z}$ in three parts, (wavenumbers are omitted)

$$\widehat{V}={\widehat{V}}_p+a{\widehat{V}}_a+b{\widehat{V}}_b,$$

where a and b are chosen in order to fulfill the homogeneous Neumann condition:

$$\begin{array}{ccc}& & \left\{\begin{array}{c}{\partial }_t{\widehat{\varphi }}_p={\widehat{h}}_v-R{e}_{\tau }^{-1}\left(k_x^2+k_z^2-{\partial }_{yy}\right){\widehat{\varphi }}_p,\hfill \\ {\widehat{\varphi }}_p\left(\pm 1\right)=0,\hfill \\ -\left(k_x^2+k_z^2-{\partial }_{yy}\right){\widehat{V}}_p={\widehat{\varphi }}_p,\hfill \\ {\widehat{V}}_p\left(\pm 1\right)=0,\hfill \end{array}\right.\hfill \end{array}$$

(21)

$$\begin{array}{ccc}& & \left\{\begin{array}{c}{\partial }_t{\widehat{\varphi }}_a=-R{e}_{\tau }^{-1}\left(k_x^2+k_z^2-{\partial }_{yy}\right){\widehat{\varphi }}_a,\hfill \\ {\widehat{\varphi }}_a\left(\pm 1\right)=0,\hfill \\ -\left(k_x^2+k_z^2-{\partial }_{yy}\right){\widehat{V}}_a={\widehat{\varphi }}_a,\hfill \\ {\widehat{V}}_a\left(1\right)=1,{\widehat{V}}_a\left(-1\right)=0,\hfill \end{array}\right.\hfill \end{array}$$

(22)

$$\begin{array}{ccc}& & \left\{\begin{array}{c}{\partial }_t{\widehat{\varphi }}_b=-R{e}_{\tau }^{-1}\left(k_x^2+k_z^2-{\partial }_{yy}\right){\widehat{\varphi }}_b,\hfill \\ {\widehat{\varphi }}_b\left(\pm 1\right)=0,\hfill \\ -\left(k_x^2+k_z^2-{\partial }_{yy}\right){\widehat{V}}_b={\widehat{\varphi }}_b,\hfill \\ {\widehat{V}}_b\left(1\right)=0,{\widehat{V}}_b\left(-1\right)=1,\hfill \end{array}\right.\hfill \end{array}$$

(23)

and

$$\left\{\begin{array}{c}{\partial }_t{\widehat{\mathsf{\Omega }}}_{y,kxkz}={\widehat{h}}_g-R{e}_{\tau }^{-1}\left(k_x^2+k_z^2-{\partial }_{yy}\right){\widehat{\mathsf{\Omega }}}_{y,kxkz},\hfill \\ {\widehat{\mathsf{\Omega }}}_y\left(\pm 1\right)=0,\hfill \end{array}\right.$$

(24)

$$\left\{\begin{array}{c}{\partial }_t{\widehat{\mathsf{\Theta }}}_{kxkz}={\widehat{h}}_t-{\left(R{e}_{\tau }Pr\right)}^{-1}\left(k_x^2+k_z^2-{\partial }_{yy}\right){\widehat{\mathsf{\Theta }}}_{kxkz},\hfill \\ {\widehat{\mathsf{\Theta }}}_{kxkz}\left(\pm 1\right)=0,\hfill \end{array}\right.$$

(25)

although, due to the symmetry of the problem, ${\widehat{V}}_b\left(y\right)={\widehat{V}}_a\left(-y\right),$ so only Equations (21), (22), (24) and (25) have to be solved. Notice that 3D PDE have been transformed into $N_x\times N_z$ independent problems.

3. Numerical Method

3.1. CFD Techniques

As it is said above, Fourier method in x and z is the best option to solve this problem. This, in fact, decouples the problems (21)–(25) in a large amount of problems in y. The numerical technique used for y is thus critical to obtain an accurate and fast algorithm. This kind of problem had been addressed mostly by spectral methods [9,31]. A typical technique used in channels is Chebyshev polynomials [11,32]. This has also been applied to thermoconvective problems [33,34,35]. However, in this case, it is more efficient the use of Compact Finite Differences, CFD. The CFD method was introduced in a groundbreaking article by SK Lele in 1992 [36]. Lele’s idea was to use finite differences to solve problems presenting a range of spatial scales, generalizing some Padé schemes that had been used early [37,38]. Lele’s CFD main advantage is that maintaining the freedom in choosing the mesh points of typical finite difference methods (FD), offering very high precision, comparable to spectral methods. This is critical in turbulent problems, as one typically needs many more points close to the wall than in the outer regions of the flow.

Throughout this part of this work, the following notation is going to be used. Let $u\left(y\right)$ be a real evaluated function. Supposing that u is differentiable enough, the first and second derivative of u at the point $y_i$ will be denoted by $u_i^{\prime }=u^{\prime }\left(y_i\right)$ and $u_i^{″}=u^{″}\left(y_i\right)$. In the case of a derivative of grade n, we will use $u_i^{n)}=\frac{d^n}{d{y}^n}u\left(y_i\right)$. Here, $y_i$ is a point belonging to a certain discretization of the interval $[a,b]$, where a and b are finite and $y_0=a$, $y_n=b$. Without loss of generality, we will focus on schemes for the second derivative. FD schemes aim to compute an approximation of $u_i^{″}$ trough a linear combination of the values of the function close to $y_i$. CFD, instead, relates a linear combination of the second derivatives with the values of the function. To explain the practical use of CFD schemes, we are going to work with one specific example, the scheme used in [29]. In that work, the authors used a stencil of seven points in the function and five in the second derivative. Let us assume that the following relation holds

$$\begin{array}{c}\hfill {\alpha }_{i-2}{u}_{i-2}^{″}+{\alpha }_{i-1}{u}_{i-1}^{″}+{\alpha }_0{u}_i^{″}+{\alpha }_{i+1}{u}_{i+1}^{″}+{\alpha }_{i+2}{u}_{i+2}^{″}\end{array}$$

(26)

$$\begin{array}{c}\hfill =a_{i-3}{u}_{i-3}+a_{i-2}{u}_{i-2}+a_{i-1}{u}_{i-1}+a_0{u}_i+a_{i+1}{u}_{i+1}+a_{i+2}{u}_{i+2}+a_{i+3}{u}_{i+3},\end{array}$$

(27)

for some unknowns coefficients, ${\alpha }_j$ and $a_j$. Without loss of generality, it is possible to assume that one of these coefficients is one, so from now on, ${\alpha }_0=1$. Defining $h_j=(y_i-y_{i+j})$, Taylor’s Theorem states that

$$\begin{array}{ccc}\hfill u_{i-3}& =& u_i+\frac{h_{-3}}{1!}{u}_i^{\prime }+\frac{h_{-3}^2}{2!}{u}_i^{″}+\cdots +\frac{h_{-3}^n}{n!}{u}_i^{n)}+O\left(h_{-3}^{n+1}\right),\hfill \\ \hfill u_{i-2}& =& u_i+\frac{h_{-2}}{1!}{u}_i^{\prime }+\frac{h_{-2}^2}{2!}{u}_i^{″}+\cdots +\frac{h_{-2}^n}{n!}{u}_i^{n)}+O\left(h_{-2}^{n+1}\right),\hfill \\ \hfill u_{i-1}& =& u_i+\frac{h_{-1}}{1!}{u}_i^{\prime }+\frac{h_{-1}^2}{2!}{u}_i^{″}+\cdots +\frac{h_{-1}^n}{n!}{u}_i^{n)}+O\left(h_{-1}^{n+1}\right),\hfill \\ \hfill u_i& =& u_i,\hfill \\ \hfill u_{i+1}& =& u_i+\frac{h_1}{1!}{u}_i^{\prime }+\frac{h_1^2}{2!}{u}_i^{″}+\cdots +\frac{h_1^n}{n!}{u}_i^{n)}+O\left(h_1^{n+1}\right),\hfill \\ \hfill u_{i+2}& =& u_i+\frac{h_2}{1!}{u}_i^{\prime }+\frac{h_2^2}{2!}{u}_i^{″}+\cdots +\frac{h_2^n}{n!}{u}_i^{n)}+O\left(h_2^{n+1}\right),\hfill \\ \hfill u_{i+3}& =& u_i+\frac{h_3}{1!}{u}_i^{\prime }+\frac{h_3^2}{2!}{u}_i^{″}+\cdots +\frac{h_3^n}{n!}{u}_i^{n)}+O\left(h_3^{n+1}\right),\hfill \end{array}$$

and

$$\begin{array}{ccc}\hfill u_{i-2}^{″}& =& u_i^{″}+\frac{h_{-2}}{1!}{u}_i^{3)}+\frac{h_{-2}^2}{2!}{u}_i^{4)}+\cdots +\frac{h_{-2}^n}{n!}{u}_i^{n+2)}+O\left(h_{-2}^{n+1}\right),\hfill \\ \hfill u_{i-1}^{″}& =& u_i^{″}+\frac{h_{-1}}{1!}{u}_i^{3)}+\frac{h_{-1}^2}{2!}{u}_i^{4)}+\cdots +\frac{h_{-1}^n}{n!}{u}_i^{n+2)}+O\left(h_{-1}^{n+1}\right),\hfill \\ \hfill u_i^{″}& =& u_i^{″},\hfill \\ \hfill u_{i+1}^{″}& =& u_i^{″}+\frac{h_1}{1!}{u}_i^{3)}+\frac{h_1^2}{2!}{u}_i^{4)}+\cdots +\frac{h_1^n}{n!}{u}_i^{n+2)}+O\left(h_1^{n+1}\right),\hfill \\ \hfill u_{i+2}^{″}& =& u_i^{″}+\frac{h_2}{1!}{u}_i^{3)}+\frac{h_2^2}{2!}{u}_i^{4)}+\cdots +\frac{h_2^n}{n!}{u}_i^{n+2)}+O\left(h_2^{n+1}\right).\hfill \end{array}$$

The relations between the coefficients $a_i$ and ${\alpha }_i$ are derived by matching the Taylor series coefficients. In this case, the formal truncation error of the approximation is tenth order. Translating this information into an algebraic equation leads to the system

$$\left(\begin{array}{ccccccccccc}0& 0& 0& 0& 1& 1& 1& 1& 1& 1& 1\\ 0& 0& 0& 0& h_{-3}& h_{-2}& h_{-1}& 0& h_1& h_2& h_3\\ -1& -1& -1& -1& \frac{h_{-3}^2}{2!}& \frac{h_{-2}^2}{2!}& \frac{1h_{-1}^2}{2!}& 0& \frac{h_1^2}{2!}& \frac{h_2^2}{2!}& \frac{h_2^2}{2!}\\ -h_{-2}& -h_{-1}& -h_1& -h_2& \frac{h_{-3}^3}{3!}& \frac{h_{-2}^3}{3!}& \frac{h_{-1}^3}{3!}& 0& \frac{h_1^3}{3!}& \frac{h_2^3}{3!}& \frac{h_3^3}{3!}\\ -h_{-2}^2& -h_{-1}^2& -h_1^2& -h_2^2& \frac{h_{-3}^4}{4·3}& \frac{h_{-2}^4}{4\times 3}& \frac{h_{-1}^4}{4\times 3}& 0& \frac{h_1^4}{4\times 3}& \frac{h_2^4}{4\times 3}& \frac{h_3^4}{4\times 3}\\ -h_{-2}^3& -h_{-1}^3& -h_1^3& -h_2^3& \frac{h_{-3}^5}{5\times 4}& \frac{h_{-2}^5}{5\times 4}& \frac{h_{-1}^5}{5\times 4}& 0& \frac{h_1^5}{5\times 4}& \frac{h_2^5}{5\times 4}& \frac{h_3^5}{5\times 4}\\ -h_{-2}^4& -h_{-1}^4& -h_1^4& -h_2^4& \frac{h_{-3}^6}{6\times 5}& \frac{h_{-2}^6}{6\times 5}& \frac{h_{-1}^6}{6\times 5}& 0& \frac{h_1^6}{6\times 5}& \frac{h_2^6}{6\times 5}& \frac{h_3^6}{6\times 5}\\ -h_{-2}^5& -h_{-1}^5& -h_1^5& -h_2^5& \frac{h_{-3}^7}{7\times 6}& \frac{h_{-2}^7}{7\times 6}& \frac{h_{-1}^7}{7\times 6}& 0& \frac{h_1^7}{7\times 6}& \frac{h_2^7}{7\times 6}& \frac{h_3^7}{7\times 6}\\ -h_{-2}^6& -h_{-1}^6& -h_1^6& -h_2^6& \frac{h_{-3}^8}{8\times 7}& \frac{h_{-2}^8}{8\times 7}& \frac{h_{-1}^8}{8\times 7}& 0& \frac{h_1^8}{8\times 7}& \frac{h_2^8}{8\times 7}& \frac{h_3^8}{8\times 7}\\ -h_{-2}^7& -h_{-1}^7& -h_1^7& -h_2^7& \frac{h_{-3}^9}{9\times 8}& \frac{h_{-2}^9}{9\times 8}& \frac{h_{-1}^9}{9\times 8}& 0& \frac{h_1^9}{9\times 8}& \frac{h_2^9}{9\times 8}& \frac{h_3^9}{9\times 8}\\ -h_{-2}^8& -h_{-1}^8& -h_1^8& -h_2^8& \frac{h_{-3}^{10}}{10\times 9}& \frac{h_{-2}^{10}}{10\times 9}& \frac{h_{-1}^{10}}{10\times 9}& 0& \frac{h_1^{10}}{10\times 9}& \frac{h_2^{10}}{10\times 9}& \frac{h_3^{10}}{10\times 9}\end{array}\right)\left(\begin{array}{c}{\alpha }_{i-2}\\ {\alpha }_{i-1}\\ {\alpha }_{i+1}\\ {\alpha }_{i+2}\\ a_{i-3}\\ a_{i-2}\\ a_{i-1}\\ a_i\\ a_{i+1}\\ a_{i+2}\\ a_{i+3}\end{array}\right)=\left(\begin{array}{c}0\\ 0\\ 1\\ 0\\ 0\\ 0\\ 0\\ 0\\ 0\\ 0\\ 0\end{array}\right).$$

Please note that this is the direct matrix coming from matching Taylor’s expansions. Usually, this matrix is ill-conditioned, so results can be very inaccurate. A typical procedure to overcome this problem consists of normalizing each row by its absolute value maximum.

As the mesh can be nonuniform, it is necessary to solve this system for each point in the mesh. When approaching the boundaries, no ghost points are used but the stencils are adapted, removing the points which lay outside the interval. This reduces the formal truncation error of the system. Once the coefficients for every point have been computed, we obtain two sparse matrices, one containing the $a_i$ coefficients, $A_{yy}$, and another one made by the ${\alpha }_i$ coefficients, $B_{yy}$, such that

$$B_{yy}{(u_0^{″},...,u_n^{″})}^t=A_{yy}{(u_0,...,u_n)}^t,$$

(28)

where the superscript ${\left(\right)}^t$ denotes the transpose. Note that (28) can be used in both ways, to derive a function or to integrate it.

3.2. Time Discretization

To get good accuracy in reasonable computation times, a third-order Runge–Kutta method has been chosen, derived in [39]. For any problem with the form

$${\partial }_t\phi =L\left(\phi \right)+N\left(\phi \right),$$

(29)

where L is a linear operator and N is nonlinear, the equation is discretized as follows:

$$\begin{array}{ccc}\hfill \left(I-{\beta }_1\mathsf{\Delta }tL\right){\phi }_1& =& \left(I+\mathsf{\Delta }t{\alpha }_{1}L\right){\phi }_0+\mathsf{\Delta }t{\gamma }_1{N}_0,\hfill \end{array}$$

(30)

$$\begin{array}{ccc}\hfill \left(I-{\beta }_2\mathsf{\Delta }tL\right){\phi }_2& =& \left(I+\mathsf{\Delta }t{\alpha }_{2}L\right){\phi }_1+\mathsf{\Delta }t\left({\gamma }_2{N}_1+{\xi }_1{N}_0\right),\hfill \end{array}$$

(31)

$$\begin{array}{ccc}\hfill \left(I-{\beta }_3\mathsf{\Delta }tL\right){\phi }_3& =& \left(I+\mathsf{\Delta }t{\alpha }_{3}L\right){\phi }_2+\mathsf{\Delta }t\left({\gamma }_3{N}_2+{\xi }_2{N}_1\right),\hfill \end{array}$$

(32)

with ${\phi }_i=\phi \left(t^i\right),$ for $t^n=t^0,t^1,t^2,t^3=t^{n+1}.$ This method presents two problems. The first one is a problem of memory, due to the necessity of saving two nonlinear terms in steps two and three. The second problem is a possible loss of accuracy near the wall, due to the explicit computation of the linear operator in the right hand,

$$L_i^t{\phi }_0=\left(I+\mathsf{\Delta }t{\alpha }_{i}L\right){\phi }_0.$$

Both problems are solved in the algorithm, using a little bit of algebra. Suppose that (30) is solved, and ${\phi }_2$ is to be computed. Let us denote

$$R{h}_1=\left(I+\mathsf{\Delta }t{\alpha }_{1}L\right){\phi }_0+\mathsf{\Delta }t{\gamma }_1{N}_0,$$

then

$$L{\phi }_1=\frac{{\phi }_1-R{h}_1}{{\beta }_1\mathsf{\Delta }t},$$

and we can compute and store in a single buffer of memory the right hand side of (31) except the nonlinear term $N_1$. This tactic can be also applied in the computation of ${\phi }_1$ but only in the cases where $\mathsf{\Delta }t$ does not change. If the time step changes, the derivative of ${\phi }_0$ has to be computed explicitly. To avoid this loss of accuracy, it is a good idea to recompute the maximum $\mathsf{\Delta }t$ every few steps, using the Courant–Friedichs–Lewy condition

$$max\left\{\frac{U\mathsf{\Delta }t}{\mathsf{\Delta }x},\frac{V\mathsf{\Delta }t}{\mathsf{\Delta }y},\frac{W\mathsf{\Delta }t}{\mathsf{\Delta }z}\right\}\le CFL.$$

The simulations made with this code ran with a CFL of $0.7,$ showing remarkable stability.

Equations (30)–(32) are solved for $\varphi$, ${\mathsf{\Omega }}_y$ and $\mathsf{\Theta }$ using CFD. In what follows we will work with the energy equation, given that the linear operator is $L=R{e}_{\tau }^{-1}{\nabla }^2$ for $\varphi$ and ${\mathsf{\Omega }}_y,$ and $L={\left(Pr{Re}_{\tau }\right)}^{-1}{\nabla }^2$ for $\mathsf{\Theta }.$ The operator L in Fourier space can be written as

$$L={\left(R{e}_{\tau }Pr\right)}^{-1}({\partial }_{xx}+{\partial }_{yy}+{\partial }_{zz})\Rightarrow {\widehat{L}}_{k_x{k}_z}={\left(R{e}_{\tau }Pr\right)}^{-1}({\left(i{k}_x\right)}^2+{\left(i{k}_z\right)}^2+{\partial }_{yy}).$$

Denoting by $RH{S}_i$ the right-hand side of (30)–(32), this system becomes

$$\left(I-{\beta }_i\mathsf{\Delta }tL\right){\phi }_i=RH{S}_i.$$

(33)

In Fourier space, one gets

$$\begin{array}{ccc}\hfill \left(I-{\beta }_i\mathsf{\Delta }t\frac{-k_x^2-k_z^2+{\partial }_{yy}}{R{e}_{\tau }Pr}\right){\phi }_i& =& RH{S}_i,\hfill \end{array}$$

(34)

$$\begin{array}{ccc}\hfill \left(I+{\beta }_i\mathsf{\Delta }t\frac{k_x^2+k_z^2}{R{e}_{\tau }Pr}\right){\phi }_i-\frac{{\beta }_i\mathsf{\Delta }t}{R{e}_{\tau }Pr}{\partial }_{yy}{\phi }_i& =& RH{S}_i,\hfill \end{array}$$

(35)

Now using the two matrix described in the CFD section (28), and defining $\eta$ as the first constant of the previous equation,

$$\eta =\left(I+{\beta }_i\mathsf{\Delta }t\frac{k_x^2+k_z^2}{R{e}_{\tau }Pr}\right),$$

we get

$$\begin{array}{ccc}\hfill \eta {\phi }_i-\frac{{\beta }_i\mathsf{\Delta }t}{R{e}_{\tau }Pr}{\partial }_{yy}{\phi }_i& =& RHS,\hfill \end{array}$$

(36)

$$\begin{array}{ccc}\hfill \eta B_{yy}{\phi }_i-\frac{{\beta }_i\mathsf{\Delta }t}{R{e}_{\tau }Pr}{B}_{yy}{\partial }_{yy}{\phi }_i& =& B_{yy}RHS,\hfill \end{array}$$

(37)

$$\begin{array}{ccc}\hfill \eta B_{yy\frac{}{}}{\phi }_i-\frac{{\beta }_i\mathsf{\Delta }t}{R{e}_{\tau }Pr}{A}_{yy}{\phi }_i& =& B_{yy}RHS.\hfill \end{array}$$

(38)

Defining a new matrix $M,$ as

$$M=\eta B_{yy}-\frac{{\beta }_i\mathsf{\Delta }t}{R{e}_{\tau }Pr}{A}_{yy},$$

(39)

the final problem to solve is

$$M{\phi }_i=B_{yy}RHS.$$

Notice that this matrix is banded, which allows low-storage schemes and the use of very efficient LAPACK routines.

4. Parallelization Strategy

The most expensive part of the algorithm is the computation of the nonlinear terms ${\widehat{h}}_v$, ${\widehat{h}}_g$ and ${\widehat{h}}_t$. This is because it is necessary to compute these terms in physical space, due to the aliasing problem [31]. Notice that there are two global operations:

• Direct and inverse Fourier transforms in x and z, as the nonlinear term has to be computed in physical space.
• Integration-derivation in y.

As the data is distributed throughout the supercomputer, it is necessary to perform several all-to-all communications, which are critical and extremely demanding of the fast network of the supercomputer. The code demands a total of $comm=3\times (9+3h_p)$ global communications per step, where $h_p$ is the total number of heat transfer problems studied.

The number of points of the problem, and thus memory, depend on the Reynolds number studied. Another constraint is the efficiency of the fast Fourier transforms (FFT), so typically numbers made up of powers of 2 and 3 are chosen to increase the velocity of the FFT. Assuming a mesh with $mx\times my\times mz$ points, generally $mx$ is the largest one. It is then natural to start each substep of the Runge–Kutta scheme with the data set distributed in y–z planes. To avoid load imbalances, the number of nodes must be a divisor of the number of complex planes, which is $mx/2$ after dealiasing. This number is also the maximum number of nodes that can be used. The maximum number of OpenMP processes at each node is $mx/2/nprocs,$ where $nprocs$ is the number of processors of the node.

The code to implement the algorithm described above, where Figure 2 can serve as roadmap, is as follows

• Read data and configuration files: HDF5
• Runge–Kutta, data: $\varphi$, ${\mathsf{\Omega }}_y$ and $\mathsf{\Theta },$ in Fourier Space, distributed through the supercomputer with a $\left(y,z,x\right)$ shape.

  (a)

  Calculation of nonlinear terms (I). Time: 15% of total time

  • $\varphi ,{\mathsf{\Omega }}_y\to \overrightarrow{U},\overrightarrow{\mathsf{\Omega }}$.
  • Compute statistical quantities of the flow if needed.
  • Change shape from $\left(y,z,x\right)$ to $\left(z,y,x\right).$
  • Perform inverse transforms in z of $\overrightarrow{U}$ and $\overrightarrow{\mathsf{\Omega }}.$

  (b)

  Calculation on nonlinear terms (II). Time: 35% of total time

  • Global transposes: Data is moved trough all the machine, from y–z planes into x-lines using MPI routines.

  (c)

  Calculation on nonlinear terms (III). Time:10% of total time

  • Perform inverse transforms in x of $\overrightarrow{U}$ and $\overrightarrow{\mathsf{\Omega }}$.
  • Computes three technical buffers $C_i$ to compute later the nonlinear terms ${\widehat{h}}_v$ and ${\widehat{h}}_g$. $h_t$ is computed here.
  • Perform direct transforms in x of $\overrightarrow{U}$ and $\overrightarrow{\mathsf{\Omega }}.$

  (d)

  Calculation on nonlinear terms (IV). Time: 20% of total time

  • Global transposes: Data is moved trough all the machine, from x-lines into y–z planes

  (e)

  Calculation on nonlinear terms (V). Time: 10% of total time

  • Perform direct transforms in z of $C_i$ and $h_t.$
  • Change shape from $\left(z,y,x\right)$ to $\left(y,z,x\right).$

  (f)

  Solve viscous problem. This step requires 10% of total time.

  (g)

  Save data to disk if needed.

  (h)

  Move to the step 2a.

• End of program.

The code spends 90% of the time and almost all communication time computing the nonlinear term and only 10% to actually solve the equations. A large share of this 90% is used to do global communications. This is a critical step and requires an in-deep study of the supercomputer where the simulation is going to run.

Finally, notice that while global communications uses MPI routines, the granularity of the code makes the use of OpenMP routines for the Fourier transforms, derivatives, and the viscous step. This is an important advantage due to the foreseen evolution of supercomputation.

5. Conclusions

The main goal of this work is to develop a code to perform direct numerical simulations of distinctive wall-bounded thermal canonical flows. This code has run for roughly 50 CPU-M hours in several supercomputers, already producing several articles. FFT and CFD techniques have been described—in particular, how to use a nonequispaced grid for CFD. In addition, the combination of the Runge–Kutta time-stepper and the previous techniques have been explained. Finally, the parallelization scheme has been outlined. Its main advantage is the granularity of the data, allowing a large amount of small problems in each step that can be solved efficiently using OpenMP techniques.

The raw data that support the findings of this study are available from the corresponding author upon reasonable request. One point statistics can be downloaded from the web page of our group: http://personales.upv.es/serhocal/ (accessed on 24 March 2021).