# Complex-Geometry 3D Computational Fluid Dynamics with Automatic Load Balancing

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Motivation and Significance

- The software implementation in Xyst enables the exploitation of the advanced features of the Charm++ runtime system with a fluid solver;
- The implementation is public and open source.

## 2. Software Description

#### 2.1. The Equations of Compressible Flow

#### 2.2. The Numerical Method

## 3. Illustrative Examples

#### 3.1. Verification

#### 3.2. Validation

#### 3.3. Automatic Load Balancing

`if fluid density > 2.0 then sleep (1 ms)`. This increases the cost of the equation of state evaluation, whose location propagates in space and time, which induces load imbalance across multiple mesh partitions in parallel. Figure 7 and Table 3 show the effect of load balancing on wall-clock time: in this particular case, the extra load would make the simulation about $36\times $ more expensive. This is made $6.1\times $ faster using load balancing.

## 4. Impact and Conclusions

## Author Contributions

## Funding

## Data Availability Statement

## Acknowledgments

## Conflicts of Interest

## References

- Luo, H.; Baum, J.D.; Löhner, R.; Cabello, J. Adaptive Edge-Based Finite Element Schemes for the Euler and Navier-Stokes Equations. In Proceedings of the 31st Aerospace Sciences Meeting, Reno, NV, USA, 1–14 January 1993. [Google Scholar]
- Luo, H.; Baum, J.D.; Löhner, R. Edge-based finite element scheme for the Euler equations. AIAA J.
**1994**, 32, 1183–1190. [Google Scholar] [CrossRef] - Waltz, J. Derived data structure algorithms for unstructured finite element meshes. Int. J. Numer. Methods Eng.
**2002**, 54, 945–963. [Google Scholar] [CrossRef] - Löhner, R.; Galle, M. Minimization of indirect addressing for edge-based field solvers. Commun. Numer. Methods Eng.
**2002**, 18, 335–343. [Google Scholar] [CrossRef] - Löhner, R. Applied Computational Fluid Dynamics Techniques: An Introduction Based on Finite Element Methods; Wiley and Sons Ltd.: Chichester, UK, 2008. [Google Scholar]
- Acun, B.; Gupta, A.; Jain, N.; Langer, A.; Menon, H.; Mikida, E.; Ni, X.; Robson, M.; Sun, Y.; Totoni, E.; et al. Parallel Programming with Migratable Objects: Charm++ in Practice. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis (SC ’14), New Orleans, LA, USA, 16–21 November 2014; pp. 647–658. [Google Scholar] [CrossRef]
- Waltz, J.; Morgan, N.; Canfield, T.; Charest, M.; Risinger, L.; Wohlbier, J. A three-dimensional finite element arbitrary Lagrangian-Eulerian method for shock hydrodynamics on unstructured grids. Comput. Fluids
**2013**, 92, 172–187. [Google Scholar] [CrossRef] - Waltz, J. Microfluidics simulation using adaptive unstructured grids. Int. J. Numer. Methods Fluids
**2004**, 46, 939–960. [Google Scholar] [CrossRef] - Davis, S.F. Simplified second-order Godunov-type methods. SIAM J. Sci. Stat. Comput.
**1988**, 9, 445–473. [Google Scholar] [CrossRef] - Hirsch, C. Numerical Computation of Internal and External Flows: The Fundamentals of Computational Fluid Dynamics; Wiley and Sons: Chichester, UK; New York, NY, USA; Brisbane, Australia; Toronto, ON, Canada; Singapore, 2007. [Google Scholar]
- Waltz, J.; Canfield, T.; Morgan, N.; Risinger, L.; Wohlbier, J. Manufactured solutions for the three-dimensional Euler equations with relevance to Inertial Confinement Fusion. J. Comput. Phys.
**2014**, 267, 196–209. [Google Scholar] [CrossRef] - Luo, H.; Baum, J.; Löhner, R. A Fast, Matrix-Free Implicit Method for Compressible Flows on Unstructured Grids. J. Comput. Phys.
**1998**, 146, 664–690. [Google Scholar] [CrossRef] - Schmitt, V.; Charpin, F. Pressure Distributions on the ONERA-M6-Wing at Transonic Mach Numbers, Experimental Data Base for Computer Program Assessment; Report of the Fluid Dynamics Panel Working Group 04; Technical rep4ort, AGARD AR-138; 1979. Available online: https://www.sto.nato.int/publications/AGARD/AGARD-AR-138/AGARD-AR-138.pdf (accessed on 27 March 2023).
- Bakosi, J.; Bird, R.; Gonzalez, F.; Junghans, C.; Li, W.; Luo, H.; Pandare, A.; Waltz, J. Asynchronous distributed-memory task-parallel algorithm for compressible flows on unstructured 3D Eulerian grids. Adv. Eng. Softw.
**2021**, 160, 102962. [Google Scholar] [CrossRef] - Li, W.; Luo, H.; Bakosi, J. A p-adaptive Discontinuous Galerkin Method for Compressible Flows using Charm++. In Proceedings of the AIAA Scitech 2020 Forum, Orlando, FL, USA, 6–10 January 2020. [Google Scholar]
- Sedov, L. Similarity and Dimensional Methods in Mechanics, 10th ed.; Elsevier: Amsterdam, The Netherlands, 1993. [Google Scholar] [CrossRef]

**Figure 1.**Initial and final velocity (

**a**,

**b**), pressure (

**c**), and total energy (

**d**) for the vortical flow problem.

**Figure 2.**${L}_{1}$ errors of the density, velocity components, and internal energy in the vortical flow problem.

**Figure 3.**Upper (

**a**) and lower (

**b**) sides of the surface mesh used for the ONERA M6 wing calculation. Computed pressure contours on the upper (

**c**) and lower (

**d**) surface.

**Figure 5.**Computed and experimental surface pressure coefficient at different semispans of the ONERA wing. Here, c denotes the length in $x={x}_{1}$ of the wing cross section at the semispan location.

**Figure 7.**Measured wall-clock time of each time step during the Sedov calculation without extra load, as well as with extra load with and without load balancing. The area below each curve is proportional to the total computational cost.

Mesh | Points | Tetrahedra | h |
---|---|---|---|

0 | 132,651 | 750,000 | 0.02 |

1 | 1,030,301 | 6,000,000 | 0.01 |

2 | 8,120,601 | 48,000,000 | 0.005 |

**Table 2.**${L}_{1}$ errors and convergence rates for the vortical flow problem. The convergence rates in the last line are computed from Equation (16) based on the errors from the 48 M and 6 M meshes.

Mesh | ${\mathit{L}}_{1}\left(\mathit{\rho}\right)$ | ${\mathit{L}}_{1}\left({\mathit{u}}_{1}\right)$ | ${\mathit{L}}_{1}\left({\mathit{u}}_{2}\right)$ | ${\mathit{L}}_{1}\left({\mathit{u}}_{3}\right)$ | ${\mathit{L}}_{1}\left(\mathit{e}\right)$ |
---|---|---|---|---|---|

750 K | $6.07\times {10}^{-5}$ | $6.68\times {10}^{-5}$ | $4.16\times {10}^{-5}$ | $7.98\times {10}^{-5}$ | $1.24\times {10}^{-3}$ |

6 M | $1.85\times {10}^{-5}$ | $1.63\times {10}^{-5}$ | $9.71\times {10}^{-6}$ | $2.00\times {10}^{-5}$ | $3.57\times {10}^{-4}$ |

48 M | $5.08\times {10}^{-6}$ | $3.93\times {10}^{-6}$ | $2.25\times {10}^{-6}$ | $4.89\times {10}^{-6}$ | $9.84\times {10}^{-5}$ |

$\mathbf{p}$ | $1.86$ | $2.05$ | $2.11$ | $2.03$ | $1.86$ |

Case | Extra Load | Total Time, s | Speed-Up |
---|---|---|---|

0 | no | 825 | - |

1 | yes | 30,276 | - |

2 | yes | 4958 | 6.11× |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Bakosi, J.; Constans, M.; Horváth, Z.; Kovács, Á.; Környei, L.; Charest, M.; Pandare, A.; Rutherford, P.; Waltz, J.
Complex-Geometry 3D Computational Fluid Dynamics with Automatic Load Balancing. *Fluids* **2023**, *8*, 147.
https://doi.org/10.3390/fluids8050147

**AMA Style**

Bakosi J, Constans M, Horváth Z, Kovács Á, Környei L, Charest M, Pandare A, Rutherford P, Waltz J.
Complex-Geometry 3D Computational Fluid Dynamics with Automatic Load Balancing. *Fluids*. 2023; 8(5):147.
https://doi.org/10.3390/fluids8050147

**Chicago/Turabian Style**

Bakosi, József, Mátyás Constans, Zoltán Horváth, Ákos Kovács, László Környei, Marc Charest, Aditya Pandare, Paula Rutherford, and Jacob Waltz.
2023. "Complex-Geometry 3D Computational Fluid Dynamics with Automatic Load Balancing" *Fluids* 8, no. 5: 147.
https://doi.org/10.3390/fluids8050147