# HPC Geophysical Electromagnetics: A Synthetic VTI Model with Complex Bathymetry

^{*}

## Abstract

**:**

## 1. Introduction

`custEM`[22],

`emg3d`[23],

`PETGEM`[24],

`SimPEG`[25], stand out. These modeling routines should be particularly sought for: (a) providing accurate solutions in a feasible runtime; (b) tackling problems efficiently; (c) bringing flexibility to cope with a variety of real-life models. The improvement of these modeling tools have a critical role in solving the next generation of geoscience challenges. These problems are complex, multidisciplinary, and require collaboration to understand and solve the physical equations, pre-process and post-process the associated data with physical experiments, and build interpretations from the analysis of the numerical results. These are the principal motivations of our work, together with the necessity for more reproducible modeling results.

`PETGEM`code to compute the synthetic EM responses, which has proven to be an efficient large-scale modeling routine on cutting-edge high-performance computing (HPC) clusters. We acknowledge that the CSEM modeling methodology presented in this paper is well-established. However, the availability of data (resistivity model, input mesh, electromagnetic responses) for reproductively/verification purposes continues to be a limiting issue in the geophysical EM modeling community. To reverse this situation, the proposed model and numerical results are based on an open approach. Furthermore, we state that our open data and benchmark are valid and useful for different application contexts (e.g., oil and gas, geothermal exploration, CO${}_{2}$ sequestration, among others) and passive-source EM schemes such as the Magnetotelluric method (MT). This transversality feature favors the expertise transfer from mature energy applications (e.g., oil and gas) to emerging energy applications (e.g., geothermal exploration).

`PETGEM`code and its HPC workflow for EM modeling. In Section 4, we provide a detailed description of the proposed marine CSEM model. In addition, we perform several

`PETGEM`simulations to investigate the EM field pattern and the code performance on HPC architectures. We conclude with a discussion in Section 5. We provide summary remarks in Section 6.

## 2. EM Modeling Theory

## 3. HPC Workflow for EM Modeling

`mpi4py`, and

`petsc4py`packages. The net result is the

`PETGEM`code, which is an EM tool to solve both active-source and passive-source EM methods in 3D arbitrary marine/land problems under anisotropic conductivities. A triple helix model based on global polynomial variants (p-refinement), tailored tetrahedral meshes (h-refinement), and massively parallel computations, stand out as the main distinguishing factors of

`PETGEM`with respect to other modeling routines.

`PETGEM`software stack is shown in Figure 1. To point out the virtues of this code environment, we summarize the essential aspects its main modules:

- Provided by user: this set of modules provides functionalities to parse user parameters regarding the physics of the EM problem to be solved (e.g., frequency and conductivity model), the solver type (e.g., direct or iterative), and the mesh file.
- Specific modules: depending on the source type (active or passive), these modules provide support to define the modeling work-flow. They are in charge of data preprocessing (e.g., import mesh file and compute its associated data structures such as dof connectivity) and data postprocessing (e.g., computation of EM responses on a set of points and output of files for posterior analysis).
- Generic modules: Once the modeling’s specifics are provided, these modules perform the assembly and solution of the linear system of Equations (LSE). These tasks are entire parallel by using the
`mpi4py`and`petsc4py`packages.

`PETGEM`code. This diagram can be summarize as follows:

- Following the input parameters, a set of data are preprocessed (mesh, conductivity model and receivers positions).
- A problem instance is created.
- The domain decomposition is performed, and the main data structures are created.
- Parallel assembling of the LSE ($Ax=b$).
- The LSE is solved in parallel by calling a ksp PETSc object.
- Interpolation of electromagnetic responses and post-processing parallel stage.

`PETGEM`code is hosted and versioned in a public on-line repository with unit-testing and continuous integration control. This open model engages users to submit and track issues, promoting the building of global communities around not only

`PETGEM`project but also for a broader audience since it fosters research at the intersection of EM modeling, high-order numerical methods, and HPC.

## 4. Numerical Experiments

- Two socket Intel Xeon Platinum 8160 CPU with 24 cores each at 2.10 GHz for a total of 48 cores per node
- L1d 32 K; L1i cache 32 K; L2 cache 1024 K; L3 cache 33,792 K
- Total of 96 GB of main memory 1.880 GB/core, 12 × 8 GB 2667 Mhz DIMM (216 nodes high memory, 10,368 cores with 7.928 GB/core)
- A 100 Gbit/s Intel Omni-Path HFI Silicon 100 Series PCI-E adapter
- A 10 Gbit Ethernet
- A 200 GB local SSD available as temporary storage during jobs

`PETGEM`website, the data model (e.g., geometry, mesh file, parameters file) and modeling results can be download for comparison purposes.

#### 4.1. 3D VTI Marine Model with Bathymetry (3D-VTI-B)

#### 4.2. EM Fields Analysis

`custEM`and

`PETGEM`solutions. A multifrontal parallel solver MUMPS has been used to solve the resulting LSE for the unknown EM fields. The MUMPS solver is supported by both EM modeling routines.

`custEM`and

`PETGEM`solutions can be seen. The synthetic EM responses have mostly a relative error of less than 1–2%. The cross-validation of the numerical approximations for non-homogeneous yields an excellent agreement with NRMSD within a few per cents. For clarity, Figure 5 shows the obtained NRMSD for amplitude and phase responses. This EM pattern is as expected and has been broadly studied in previous works for several and different CSEM setups [28,29]. Nonetheless, we acknowledge that the electric field pattern depends on the input model setup (e.g., source type, fundamental frequency, resistivity, depth of station where EM fields are measured). For a comprehensive analysis of the impact of these factors on EM field behaviour we refer to [32].

#### 4.3. Performance Analysis

`PETGEM`code for the solution of the proposed 3D-VTI-B model. For this test, we solve five fundamental frequencies on adapted meshes for basis order $p=2$. The obtained grid statistics are summarized in Table 1. We state that the design of adapted meshes can be advantageous for the solution of the CSEM test under study. A close assessment of Table 1 shows that the number of tetrahedal elements and dof in the mesh remains constant for all frequencies. Further, the run-time to reach the solution for each fundamental frequency is almost constant. The iterative

`GMRES`solver provided by

`PETSc`has been used to solve the LSE.

`PETSc`solver implementations, no particular work was undertaken to reduce run-time, as this is an completely different goal. However, these numerical results indicate a remarkable benefit when parallel executions are employed (similar conclusions to that described in [31,32]).

## 5. Discussion

`custEM`and

`PETGEM`. The main purpose of the first test is to study the impact of the reservoir presence on the measured EM responses. The cross-validation between

`custEM`and

`PETGEM`yields a similar EM pattern (the obtained synthetic EM responses have mostly a relative misfit of less than 1–3%). Thus, we consider these numerical results correct because comparing different modeling routines that use different numerical schemes is proper to address the topic of verification.

`GMRES`solver to compute the numerical solutions for the introduced resistivity model. In general, a considerable run-time improvement can be seen by performing a strong scaling test. We state that the improvement of reported run-time has been achieved through HPC, a differentiating

`PETGEM`feature concerning the rest of EM modeling tools. Our performance study shows that high-frequencies achieve better performance ratios than low-frequencies.

## 6. Conclusions

`PETGEM`code offers excellent computational performance ratios. The obtained performance metrics are consistent with previously published results. Furthermore, our previously published tailored gridding strategy has been proved in the presence of complex bathymetry and for modeling different fundamental frequencies. Our mesh design approach demonstrated that can deal with challenging bathymetry and realistic physical parameters.

`PETGEM`code features fulfill modeling requisites of practical setups in the context of HPC geophysical electromagnetics. We acknowledge that validating and testing 3D algorithms and codes are difficult tasks. Then, it is fundamental to have benchmark models that are developed under an open-source scheme that promotes easy access to data and reproducible solutions. Therefore, in the corresponding

`PETGEM`website and Zenodo, the dataset and all results can be download for comparison purposes. We trust that these numerical dataset may be helpful for the geophysical community interested in HPC geoelectromagnetics. We stir the community to design more models, modeling tools, and numerical experiments under an open-source approach.

## Author Contributions

`PETGEM`simulations, Wrote the manuscript. J.d.l.P.: Analyzed and interpreted the modeling results, Super vised the overall study, Critically revised the manuscript. J.M.C.: Analyzed and interpreted the modeling results, Critically revised the manuscript. All authors have read and agreed to the published version of the manuscript.

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Data Availability Statement

`PETGEM`code for the 3D-VTI-B model, as well as all files to rerun the model with the

`PETGEM`and reproduce the shown results are available at Zenodo (10.5281/zenodo.5842270). The

`PETGEM`code is freely available at the home page project (https://petgem.bsc.es), at the PyPI repository (https://pypi.org/project/petgem), at the GitHub site (https://github.com/ocastilloreyes/petgem), or by requesting it from the author ([email protected]). In all cases, the code is supplied to ease the immediate execution on Linux platforms. User’s manual and technical documentation (developer’s guide) are provided in the

`PETGEM`archive.

## Acknowledgments

`PETGEM`has received funding from the European Union’s Horizon 2020 programme, grant agreement N${}^{\circ}$ 828947, and from the Mexican Department of Energy, CONACYT-SENER Hidrocarburos grant agreement N${}^{\circ}$ B-S-69926. We would like to thank Jazmin Ester Gell for plotting of block diagram overview of

`PETGEM`.

## Conflicts of Interest

## References

- Castillo-Reyes, O.; Modesto, D.; Queralt, P.; Marcuello, A.; Ledo, J.; Amor-Martin, A.; de la Puente, J.; García-Castillo, L.E. 3D magnetotelluric modeling using high-order tetrahedral Nédélec elements on massively parallel computing platforms. Comput. Geosci.
**2022**, 1, 105030. [Google Scholar] [CrossRef] - Raiche, A.P. An integral equation approach to three-dimensional modelling. Geophys. J. Int.
**1974**, 1, 363–376. [Google Scholar] [CrossRef] [Green Version] - Hursán, G.; Zhdanov, M.S. Contraction integral equation method in three-dimensional electromagnetic modeling. Radio Sci.
**2002**, 1, 1–13. [Google Scholar] [CrossRef] [Green Version] - Zhdanov, M.S.; Lee, S.K.; Yoshioka, K. Integral equation method for 3D modeling of electromagnetic fields in complex structures with inhomogeneous background conductivity. Geophysics
**2006**, 71, G333–G345. [Google Scholar] [CrossRef] - Tehrani, A.M.; Slob, E. Fast and accurate three-dimensional con- trolled source electromagnetic modelling. Geophys. Prospect.
**2010**, 58, 1133–1146. [Google Scholar] - Kruglyakov, M.; Bloshanskaya, L. High-performance parallel solver for integral equations of electromagnetics based on Galerkin method. Math. Geosci.
**2016**, 49, 751–776. [Google Scholar] [CrossRef] [Green Version] - Wang, T.; Hohmann, G.W. A finite-difference, time-domain solution for three-dimensional electromagnetic modeling. Geophysics
**1993**, 58, 797–809. [Google Scholar] [CrossRef] [Green Version] - Druskin, V.; Knizhnerman, L. Spectral approach to solving three-dimensional Maxwell’s diffusion equations in the time and frequency domains. Radio Sci.
**1994**, 1, 937–953. [Google Scholar] [CrossRef] - Mackie, R.L.; Smith, J.T.; Madden, T.R. Three-dimensional electro-magnetic modeling using finite difference equations: The magnetotelluric example. Radio Sci.
**1994**, 29, 923–935. [Google Scholar] [CrossRef] - Streich, R. 3D finite-difference frequency-domain modeling of controlled-source electromagnetic data: Direct solution and optimization for high accuracy. Geophysics
**2009**, 74, F95–F105. [Google Scholar] [CrossRef] - Commer, M.; Newman, G. A parallel finite-difference approach for 3D transient electromagnetic modeling with galvanic sources. Geophysics
**2004**, 69, 1192–1202. [Google Scholar] [CrossRef] [Green Version] - Schwarzbach, C.; Börner, R.-U.; Spitzer, K. Three-dimensional adaptive higher order finite element simulation for geoelectromagnetics—A marine CSEM example. Geophys. J. Int.
**2011**, 1, 63–74. [Google Scholar] [CrossRef] [Green Version] - da Silva, N.V.; Morgan, J.V.; MacGregor, L.; Warner, M. A finite element multifrontal method for 3D CSEM modeling in the frequency domain. Geophysics
**2012**, 77, E101–E115. [Google Scholar] [CrossRef] - Grayver, A.V.; Streich, R.; Ritter, O. Three-dimensional parallel distributed inversion of CSEM data using a direct forward solver. Geophys. J. Int.
**2013**, 193, 1432–1446. [Google Scholar] [CrossRef] [Green Version] - Zhang, Y.; Key, K. MARE3DEM: A three-dimensional CSEM inversion based on a parallel adaptive finite element method using unstructured meshes. In SEG Technical Program Expanded Abstracts; Onepetro: Dallas, TX, USA, 2016; pp. 1009–1013. [Google Scholar]
- Madsen, N.K.; Ziolkowski, R.W. A three-dimensional modified finite volume technique for Maxwell’s equations. Electromagnetics
**1990**, 10, 147–161. [Google Scholar] [CrossRef] - Clemens, M.; Weiland, T. Discrete electromagnetism with the finite integration technique. PIER
**2001**, 32, 65–87. [Google Scholar] [CrossRef] [Green Version] - Jahandari, H.; Farquharson, C.G. A finite-volume solution to the geophysical electromagnetic forward problem using unstructured grids. Geophysics
**2014**, 79, E287–E302. [Google Scholar] [CrossRef] - Newman, G.A.; Commer, M.; Carazzone, J.J. Imaging CSEM data in the presence of electrical anisotropy. Geophysics
**2010**, 75, F51–F61. [Google Scholar] [CrossRef] - Sommer, M.; Hölz, S.; Moorkamp, M.; Swidinsky, A.; Heincke, B.; Scholl, C.; Jegen, M. GPU parallelization of a three dimensional marine CSEM code. Comput. Geosci.
**2013**, 58, 91–99. [Google Scholar] [CrossRef] - Hong, C.; Sukumaran-Rajam, A.; Nisa, I.; Singh, K.; Sadayappan, P. Adaptive sparse tiling for sparse matrix multiplication. In Proceedings of the 24th Symposium on Principles and Practice of Parallel Programming, Washington, DC, USA, 16–20 February 2019; pp. 300–314. [Google Scholar]
- Rochlitz, R.; Skibbe, N.; Günther, T. custEM: Customizable finite element simulation of complex controlled-source electromagnetic data. Geophysics
**2019**, 84, F17–F33. [Google Scholar] [CrossRef] - Werthmüller, D. An open-source full 3D electromagneticmodeler for 1D VTI media in Python: Empymod. Geophysics
**2017**, 82, WB9–WB19. [Google Scholar] [CrossRef] - Castillo-Reyes, O.; de la Puente, J.; Cela, J.M. PETGEM: A parallel code for 3D CSEM forward modeling using edge finite elements. Comput. Geosci.
**2018**, 119, 123–136. [Google Scholar] [CrossRef] [Green Version] - Heagy, L.J.; Cockett, R.; Kang, S.; Rosenkjaer, G.K.; Olden-burg, D.W. A framework for simulation and inversion in electromagnetics. Comput. Geosci.
**2017**, 107, 1–19. [Google Scholar] [CrossRef] - Zhdanov, M.S. Geophysical Electromagnetic Theory and Methods; Elsevier: Amsterdam, The Netherlands, 2009; Volume 43, ISBN 9780444529633. [Google Scholar]
- Nam, M.J.; Kim, H.J.; Song, Y.; Lee, T.J.; Son, J.; Suh, J. 3D magnetotelluric modelling including surface topography. Geophys. Prospect.
**2007**, 55, 277–287. [Google Scholar] [CrossRef] - Constable, S. Ten years of marine CSEM for hydrocarbon exploration. Geophysics
**2010**, 5, 75A67–75A81. [Google Scholar] [CrossRef] [Green Version] - Kerry, K.; Ovall, J. A parallel goal-oriented adaptive finite element method for 2.5-D electromagnetic modelling. Geophys. J. Int.
**2011**, 1, 137–154. [Google Scholar] - Jin, J. The Finite Element Method in Electromagnetics, 2nd ed.; Wiley: New York, NY, USA, 2002. [Google Scholar]
- Castillo-Reyes, O.; de la Puente, J.; García-Castillo, L.E.; Cela, J.M. Parallel 3-D marine controlled-source electromagnetic modelling using high-order tetrahedral Nédélec elements. Geophys. J. Int.
**2019**, 1, 39–65. [Google Scholar] [CrossRef] - Castillo-Reyes, O.; Queralt, P.; Marcuello, A.; Ledo, J. Land CSEM Simulations and Experimental Test Using Metallic Casing in a Geothermal Exploration Context: Vallès Basin (NE Spain) Case Study. IEEE Trans. Geosci. Remote Sens.
**2021**, 60, 1–13. [Google Scholar] [CrossRef]

**Figure 3.**Synthetic 3D VTI marine model with bathymetry. A 3D view of the complex seabed and the reservoir unit are provided. The resistivity values for each material are also given.

**Figure 4.**Electric field responses for in-line profile of the synthetic 3D-VTI-B model depicted in Figure 3. Details of the seabed profile with transmitter and receivers positions are depicted in the top row. The amplitude $|{\mathbf{E}}_{x}|$ and phase ${\varphi}_{x}$ are depicted in the middle and bottom row, respectively. Homogeneous and non-homogeneous medium are compared.

**Figure 5.**NRMSD for amplitude $|{\mathbf{E}}_{x}|$ and phase ${\varphi}_{x}$ depicted in Figure 4. Non-homogeneous setup of 3D-VTI-B model is compared.

**Figure 6.**Electric field amplitude response of the 3D-VTI-B model around the reservoir unit. Three view profiles of the vector electric field components are given.

**Figure 7.**Performance results for the 3D-VTI-B model. For each fundamental frequency, the number of CPU N is plotted versus speed-up S (

**left**-panel) and parallel efficiency ratio E (

**right**-panel). The solid red line shows the theoretical ideal performance supposing 100% parallel efficiency.

**Table 1.**Statistics of adapted meshes and performance results for 3D-VTI-B model. The frequency (Hz), number of elements, number of dof, run-time (minutes), speed-up S, and parallel efficiency E (%) for five relevant simulations are given.

Frequency | Elements | Dof | Run-Time | S | E | ||||||
---|---|---|---|---|---|---|---|---|---|---|---|

48 | 528 | 1008 | 48 | 528 | 1008 | 48 | 528 | 1008 | |||

10 | 602,593 | 3,678,324 | 88.85 | 9.06 | 4.99 | - | 9.8 | 17.79 | - | 89.09 | 84.76 |

5.0 | 615,378 | 3,737,031 | 76.85 | 8.35 | 4.54 | - | 9.2 | 16.90 | - | 83.63 | 80.47 |

2.5 | 611,482 | 3,714,628 | 68.37 | 7.51 | 4.19 | - | 9.1 | 16.29 | - | 82.72 | 77.61 |

1.0 | 613,247 | 3,721,364 | 61.45 | 7.02 | 4.12 | - | 8.74 | 14.89 | - | 79.54 | 70.95 |

0.5 | 608,577 | 3,864,769 | 49.78 | 6.10 | 3.61 | - | 8.15 | 13.77 | - | 74.09 | 65.61 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Castillo-Reyes, O.; de la Puente, J.; Cela, J.M.
HPC Geophysical Electromagnetics: A Synthetic VTI Model with Complex Bathymetry. *Energies* **2022**, *15*, 1272.
https://doi.org/10.3390/en15041272

**AMA Style**

Castillo-Reyes O, de la Puente J, Cela JM.
HPC Geophysical Electromagnetics: A Synthetic VTI Model with Complex Bathymetry. *Energies*. 2022; 15(4):1272.
https://doi.org/10.3390/en15041272

**Chicago/Turabian Style**

Castillo-Reyes, Octavio, Josep de la Puente, and José María Cela.
2022. "HPC Geophysical Electromagnetics: A Synthetic VTI Model with Complex Bathymetry" *Energies* 15, no. 4: 1272.
https://doi.org/10.3390/en15041272