Heterogeneous CPU-GPU Accelerated Subgridding in the FDTD Modelling of Microwave Breakdown

Abstract

Microwave breakdown is crucial to the transmission of high-power microwave (HPM) devices, where a growing number of studies have analyzed the complex interactions between electromagnetic waves and the evolving plasma from theoretical and analytical perspectives. In this paper, we propose a finite-difference time-domain (FDTD) scheme to numerically solve Maxwell’s equation, coupled with a fluid plasma equation for simulating the plasma formation during HPM air breakdown. A subgridding method is adopted to obtain accurate results with lower computational resources. Moreover, the three-dimensional subgridding Maxwell–plasma algorithm is efficiently accelerated by utilizing heterogeneous computing technique based on graphics processing units (GPUs) and multiple central processing units (CPUs), which can be applied as an efficient method for the investigation of the HPM air breakdown phenomena.

1. Introduction

High-power microwave (HPM) systems and devices are widely used in the civilian and military sectors. To better design HPM systems and devices, air breakdown using a HPM source has been extensively investigated and is relatively well understood experimentally [1,2,3,4], as well as theoretically [5,6,7,8,9,10]. In the research on HPM, air breakdown is essential because of the significant limiting factor in the transmission of HPM devices. The transmission capability of a device will be dramatically decreased if the breakdown occurs.

Numerical analyses of microwave breakdown in metallic holes have been well studied. By coupling Maxwell’s equations and a set of electron-fluid model, the effect of air breakdown on intense microwave pulses passing through enclosed aperture holes was studied [11]. A similar conclusion of microwave breakdown in a narrow metallic gap was also obtained theoretically and experimentally in [12]. In [9,10], the FDTD has been adopted to solve a set of coupled Maxwell–plasma problems. Because of the relatively large plasma density gradient during microwave breakdown, the cell size should be small enough to achieve an accurate solution. According to the Courant–Friedrichs–Lewy (CFL) limit, the time step will also be small. As mentioned above, this uniform grid settings hinders the FDTD method from being applied to microwave breakdown problems because of the huge memory requirement and the long execution time. Due to the issues mentioned above, Hamiaz et al. proposed a 2D Maxwell–plasma model based on the finite-volume time-domain (FVTD) scheme, and it is suitable to study microwave breakdown in complex geometries [13]. The results prove that in the plasma region the FVTD method allows for local mesh refinement. A time step of local region used in the FVTD allows for a reduction of the required computational time. In [14,15], the discontinuous Galerkin time-domain (DGTD) [16,17,18] method was proposed for the multiscale and multiphysics modeling and simulation of the HPM breakdown and the Maxwell-plasma interaction. The nodal DGTD method was applied in the solution of the parabolic diffusion equation and the hyperbolic Maxwell’s equations.

In this paper, subgridding FDTD that is accelerated by heterogeneous computing techniques based on GPUs and multiple CPUs is applied to study the air breakdown in HPM devices, which shows significant advantages over conventional FDTD. In the first part of Section 2, the physical model is presented, which is similar to the one used in [19]. The three-dimensional numerical framework is also introduced in this section. In the second part of Section 2, we present the subgridding algorithm and heterogeneous CPU-GPU acceleration to improve computational efficiency and save memory. In Section 3, the subgridding algorithm is used to simulate the air breakdown in the high-power microwave device, which is accelerated by heterogeneous CPU-GPU computing technique. It is found that the simulation results are in good agreement with the experimental results. In Section 4, a discussion on microwave breakdown and acceleration technique is concluded, as an attempt to understand the multiphysics and multiscale problems.

2. Physical and Computation Model

2.1. Physical Model

The physical model adopted in this paper is based on standard Maxwell’s equations linked with a simple plasma diffusion equation [20,21,22,23]. The electromagnetic fields $E$ and $H$ drive the motion of the plasma charge with electron density $n_e$, velocity $v_e$ and current density $J$. The physical model is given by

$$\nabla \times H=J+{\epsilon }_0\frac{\partial E}{\partial t}$$

(1)

$$\nabla \times E=-{\mu }_0\frac{\partial H}{\partial t}$$

(2)

$$J=-e{n}_e{v}_e$$

(3)

where $e$ is the charge of the electron. The ion current density is neglected because it is much smaller compared with the electron current density. In Equation (3), the electron mean velocity $v_e$ is the solution of the simplified momentum equation

$$\frac{\partial v_e}{\partial t}=-\frac{e}{m_e}E-v_c{v}_e$$

(4)

where $m_e$ is the electron mass and the $v_c$ is the electron-neutral collision frequency given by

$$v_c=5.3\times {10}^{9}P$$

(5)

where the $P$ is the ambient pressure in Torr. In this paper, $P=100$ Torr. The electron density $n_e$ is governed by a mass conservation equation

$$\frac{\partial n_e}{\partial t}-\nabla ·(D_{eff}\nabla n_e)=(v_i-v_a)n_e-r_{ei}{n}_e{}^2$$

(6)

where the operator $\nabla ·(D_{eff}\nabla n_e)$ can be reformulated as the sum of two operators

$$\nabla ·(D_{eff}\nabla n_e)=\nabla D_{eff}·\nabla n_e+D_{eff}{\nabla }^2{n}_e$$

(7)

According to [24], the first term on the right ($\nabla D_{eff}·\nabla n_e$) in Equation (7) and the second term on the right ($r_{ei}{n}_e{}^2$) in Equation (6) are insignificant and can be neglected.

The effective diffusion coefficient $D_{eff}$ can be written as a combination of the electron free diffusion coefficient $D_e$ and the ambipolar diffusion coefficient $D_e$ as follows

$$D_{eff}=\frac{\alpha D_e+D_a}{\alpha +1}$$

(8)

where

$$D_e={\mu }_e\frac{k_B{T}_e}{e} ({\mathrm{m}}^2/\mathrm{s})$$

(9)

$$D_a\approx \frac{{\mu }_i}{{\mu }_e}{D}_e\begin{array}{cc}& ({\mathrm{m}}^2/\mathrm{s})\end{array}$$

(10)

where $k_B$ is the Boltzmann constant, $T_e$ is the electron temperature which can be considered as approximately constant and equal to 2 eV [19], and ${\mu }_e$ stands for the electron mobility, which is equal to $e/m_e{v}_c$. The ion mobility is usually supposed to be ${\mu }_i={\mu }_e/200 ({\mathrm{m}}^2/\mathrm{V}\cdot \mathrm{s})$ [24]. The weighting between the free and ambipolar diffusions is adjusted by the unitless factor $\alpha$ as

$$\alpha =v_i{\tau }_M$$

(11)

where ${\tau }_M={\epsilon }_0/[e{n}_e({\mu }_e+{\mu }_i)]$ represents the local dielectric relaxation time.

In Equation (6), the effective ionization frequency usually replaced the combination of attachment frequency $v_a$ and ionization frequency $v_i$, which can be represented by the electron drift velocity $v_d={\mu }_{e}E$ [25,26] as

$$v_{\mathit{eff}}=v_i-v_a=\gamma v_d$$

(12)

where $\gamma$ is the ionization coefficient and indicated by the following empirical expression

$$\gamma =\{\begin{array}{l}{A}_{0}p[e^{-B_0(p/E_{eff}-p/E_c)}-1] (\mathrm{Hz})\hspace{1em}\hspace{1em}{E}_{eff}/p<50 (\mathrm{V}/\mathrm{cm}·\mathrm{torr})\\ Ap{e}^{-Bp/E_{eff}}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}\hspace{1em}(\mathrm{Hz})\hspace{1em}\hspace{1em}{E}_{eff}/p\ge 50 (\mathrm{V}/\mathrm{cm}·\mathrm{torr})\end{array}$$

(13)

where $A_0=0.05{ (\mathrm{cm}}^{-1}{\mathrm{torr}}^{-1})$, $B_0=200 (\mathrm{V}/\mathrm{cm}·\mathrm{torr})$, and $E_c/p=31.25 (\mathrm{V}/\mathrm{cm}·\mathrm{torr})$, $E_c$ is the critical field intensity which determines the proportion of ionization and attachment. The coefficients are $A=8.805{ (\mathrm{cm}}^{-1}{\mathrm{torr}}^{-1})$, $B=258.45 (\mathrm{V}/\mathrm{cm}·\mathrm{torr})$ when $50\le E_{eff}/p\le 200 (\mathrm{V}/\mathrm{cm}·\mathrm{torr})$ and $A=15{ (\mathrm{cm}}^{-1}{\mathrm{torr}}^{-1})$, and $B=265 (\mathrm{V}/\mathrm{cm}·\mathrm{torr})$ when $100\le E_{eff}/p\le 800 (\mathrm{V}/\mathrm{cm}·\mathrm{torr})$ [25].

In Equation (13), the effective field $E_{eff}$ represented by

$$E_{eff}=\frac{E_{rms}}{\sqrt{1+\frac{{\omega }^2}{v_c}}}$$

(14)

where the mean square root $E_{rms}$ is defined as

$$E_{rms}(m,n,p)=\sqrt{{\left(\frac{E_x[m-\frac{1}{2},n,p]+E_x[m+\frac{1}{2},n,p]}{2}\right)}^2+{\left(\frac{E_y[m,n-\frac{1}{2},p]+E_y[m,n+\frac{1}{2},p]}{2}\right)}^2+{\left(\frac{E_z[m,n,p-\frac{1}{2}]+E_z[m,n,p+\frac{1}{2}]}{2}\right)}^2}$$

(15)

We discretized our domain via the Yee grid [27], where the electric fields are collocated with the electron mean velocity $v_e$ in space, the electron density $n_e$ is collocated at the vertices of the Yee grid. The electron velocity $v_e$ and electron density $n_e$ are at the same time step with that of the magnetic fields. The computational grid based on the standard Yee cell is displayed in Figure 1.

Considering the electron mean velocity $v_e$ at $t\Delta t$, the explicit difference equation for $v_e$ can be deduced as

$$\frac{v_e^{t+\frac{1}{2}}-v_e^{t-\frac{1}{2}}}{\Delta t}=-\frac{e}{m_e}{E}^t-v_c\frac{v_e^{t+\frac{1}{2}}+v_e^{t-\frac{1}{2}}}{2}$$

(16)

with

$$v_e^{t+\frac{1}{2}}=\left\{\begin{array}{c}{v}_{ex}^{t+\frac{1}{2}}[m+\frac{1}{2},n,p]\\ v_{ey}^{t+\frac{1}{2}}[m,n+\frac{1}{2},p]\\ v_{ez}^{t+\frac{1}{2}}[m,n,p+\frac{1}{2}]\end{array}\right\}\begin{array}{cc}& \end{array}\mathrm{and}\begin{array}{cc}& \end{array}{E}_e^t=\left\{\begin{array}{c}{E}_x^t[m+\frac{1}{2},n,p]\\ E_y^t[m,n+\frac{1}{2},p]\\ E_z^t[m,n,p+\frac{1}{2}]\end{array}\right\}$$

(17)

with the central spatial difference and with the explicit time-stepping, the discretized form of Equation (6) is shown as follows:

$$\begin{array}{l}{n}_e^{t+\frac{1}{2}}[m,n,p]=n_e^{t-\frac{1}{2}}[m,n,p]+\Delta t{D}_{eff}(\frac{1}{\Delta x^2}(n_e^{t-\frac{1}{2}}[m+1,n,p]-2n_e^{t-\frac{1}{2}}[m,n,p]+n_e^{t-\frac{1}{2}}[m-1,n,p])\\ +\frac{1}{\Delta y^2}(n_e^{t-\frac{1}{2}}[m,n+1,p]-2n_e^{t-\frac{1}{2}}[m,n,p]+n_e^{t-\frac{1}{2}}[m,n-1,p])\\ +\frac{1}{\Delta z^2}(n_e^{t-\frac{1}{2}}[m,n,p+1]-2n_e^{t-\frac{1}{2}}[m,n,p]+n_e^{t-\frac{1}{2}}[m,n,p-1])+\Delta t(v_i-v_a)n_e^{t-\frac{1}{2}}[m,n,p])\end{array}$$

(18)

2.2. Subgridding Algorithm and Heterogeneous CPU-GPU Acceleration

For the traditional FDTD method using the pure dense grid, especially for the simulation of EM interacting with plasmas and targets with delicate structures [28,29,30], the grid size must be small enough to capture the motion of electromagnetic fields and plasma dynamics. However, the fine mesh for simulating plasmas will lead to large memory cost and a long execution time. To solve this problem, the subgridding mesh technique is adopted [31,32,33]. Figure 2 shows the 3-D subgridding interface and the x-component electric field interpolation schematic diagram in the x–z plane. For the sake of simplicity, a grid ratio (GR) of the coarse grid and the dense grid is 3:1 is illustrated. $‘E’$ represents the electric fields in the coarse grid and $‘e’$ indicates the electric fields and in the dense grid. There are three different interpolation types in Figure 2. $e_{x2}$ and $e_{x5}$ belong to type one, in which the coarse grid electric fields overlap with dense grid electric fields, and these electric fields can be directly obtained from coarse grid electric fields. $e_{x1}$, $e_{x3}$, $e_{x4}$ and $e_{x6}$ belong to type two, in which these electric fields can be obtained with the linear interpolation method using the coarse grid electric fields. $e_{x7}$, $e_{x8}$, $e_{x9}$, $e_{x10}$, $e_{x11}$ and $e_{x12}$ belong to type three, in which these electric fields can be obtained with the linear interpolation method using the dense grid electric fields. The coefficients are displayed in Figure 2b.

The update equations for coarse grid electric fields in the coarse grid region, such as $E_{x1}$ and $E_{x4}$, are those of the standard FDTD. The update equation for coarse grid electric fields at the interface, such as $E_{x2}$, $E_{x3}$, $E_{x5}$ and $E_{x6}$, is given by

$$\begin{array}{l}{E}_x^{q+1}(m,n,p)=E_x^q(m,n,p)\\ +(\frac{\Delta t}{\epsilon (\frac{\varsigma +1}{2\varsigma }\Delta y)}({\displaystyle \sum _{k=1}^{{(2\varsigma -1)}^2}{C}_k\times h_{{{}_z}_k}^{q+\frac{1}{2}}-}{H}_z^{q+\frac{1}{2}}(m,n-1,p))-\frac{\Delta t}{\epsilon \Delta z}(H_y^{q+\frac{1}{2}}(m,n,p)-H_y^{q+\frac{1}{2}}(m,n,p-1)))\end{array}$$

(19)

where $\varsigma$ is the grid ratio and $h_{zk}$ are the magnetic fields in the dense grid region. There are four types of dense grid magnetic fields at the interface, as shown in Figure 3. Most of the dense grid magnetic fields belong to the type in Figure 3a, in which the electric field is at the middle of the interface and the magnetic fields are all in the dense grid region. In Figure 3b, the electric field is parallel to the boundary of the interface and there are $2\varsigma -1$ dense magnetic fields in x-direction, but only $\varsigma -1$ in the z-direction. In Figure 3c, the electric field is perpendicular to the boundary of the interface and there are $2\varsigma -1$ dense magnetic fields in the z-direction, but only $\varsigma +(\varsigma -1)/2$ in the x-direction. In Figure 3d, the electric field is perpendicular to one boundary of interface and parallel to another boundary of the interface, where there are $\varsigma -1$ dense magnetic fields in the z-direction and $\varsigma +(\varsigma -1)/2$ in the x-direction.

The coefficient matrix $C_k$ in Equation (19) has different four expressions which correspond to four different situations in Figure 3.

$C=\left[\begin{array}{ccccccc}1& 2& \cdots & \varsigma & \cdots & 2& 1\\ 2& 2^2& & 2\varsigma & & 2^2& 2\\ ⋮& & \ddots & ⋮& ⋰& & ⋮\\ \varsigma & 2\varsigma & \cdots & {\varsigma }^2& \cdots & 2\varsigma & \varsigma \\ ⋮& & ⋰& ⋮& \ddots & & ⋮\\ 2& 2^2& & 2\varsigma & & 2^2& 2\\ 1& 2& \cdots & \varsigma & \cdots & 2& 1\end{array}\right]/{\varsigma }^4$	$C=\left[\begin{array}{ccccccc}1& 2& \cdots & \varsigma & \cdots & 2& 1\\ 2& 2^2& & 2\varsigma & & 2^2& 2\\ ⋮& & \ddots & ⋮& ⋰& & ⋮\\ \varsigma -1& 2(\varsigma -1)& \cdots & \varsigma (\varsigma -1)& \cdots & 2(\varsigma -1)& \varsigma -1\end{array}\right]/({\varsigma }^3\times \frac{\varsigma -1}{2})$
(a)	(b)
$C=\left[\underset{(3\varsigma -1)/2}{\underbrace{\begin{array}{ccccccc}\varsigma & \cdots & \varsigma & \varsigma & \cdots & 2& 1\\ 2\varsigma & \cdots & 2\varsigma & 2\varsigma & & 2^2& 2\\ ⋮& & & ⋮& ⋰& & ⋮\\ {\varsigma }^2& \cdots & {\varsigma }^2& {\varsigma }^2& \cdots & 2\varsigma & \varsigma \\ ⋮& & & ⋮& \ddots & & ⋮\\ 2\varsigma & \cdots & 2\varsigma & 2\varsigma & & 2^2& 2\\ \varsigma & \cdots & \varsigma & \varsigma & \cdots & 2& 1\end{array}}}\right]/{\varsigma }^4$	$C=\left[\underset{(3\varsigma -1)/2}{\underbrace{\begin{array}{ccccccc}\varsigma & \varsigma & \cdots & \varsigma & \cdots & 2& 1\\ 2\varsigma & 2\varsigma & & 2\varsigma & & 2^2& 2\\ ⋮& & \ddots & ⋮& ⋰& & ⋮\\ \varsigma (\varsigma -1)& \varsigma (\varsigma -1)& \cdots & \varsigma (\varsigma -1)& \cdots & 2(\varsigma -1)& \varsigma -1\end{array}}}\right]/({\varsigma }^3\times \frac{\varsigma -1}{2})$
(c)	(d)

The other five interfaces follow the same linear interpolation method. The subgridding algorithm, with different GR values, which are equal to 5, 7, and 9 are also employed in Maxwell–plasma equations in this paper.

Generally, a dense mesh and coarse mesh are adopted for updating plasmas and low refractive index media, respectively. As a rule-of-thumb, the dense/coarse grid size should be set to at least 200/10 cells per wavelength for the plasma and dielectrics. To efficiently execute the FDTD code, we used a heterogeneous computing technique. An example of a heterogeneous computing platform is shown in Figure 4, containing at least one multicore CPU and one or more GPUs. For the computation of EM fields in coarse grid regions, the CPU is employed, whereas the GPU is employed for the computation of EM fields and physical fields within dense grid regions. The PCI-e port connects the GPUs with the host. Different programming models and vendor-specific techniques are incorporated into this system to ensure cross-device execution of applications. In addition, this system provides kernel launch control, automatic data communication mechanisms, and memory management of devices. A GPU is well suited to parallel computation due to its Compute Unified Device Architecture (CUDA), which provides an interface for multiple languages and is compatible with several compilers. An abstracted representation of the GPU’s hard resources is called a 3-D grid, whose basic unit is called a block. The block is a 3-D structure made up of threads, which constitute its basic unit. In parallel programs, the scheduler assigns tasks within a block to a stream multiprocessor (SM), with each SM handling multiple blocks simultaneously.

In this paper, there are two types of parallelism, one is the coarse-grained parallelism and the other is the fine-grained parallelism. In a heterogeneous system, the parallelism between CPUs and GPU is coarse-grained parallelism. This type of parallelism is achieved by arranging the coarse grid electric fields and magnetic fields in CPUs and arranging the dense grid electric fields, magnetic fields, electron density, and velocity in GPUs. The parallelism between threads is fine-grained parallelism. As shown in Figure 5, the FDTD grid is decomposed and allocated to the different processor units (PUs) in the heterogeneous system. Consequently, the components on the truncated boundary, as well as those involved in their updated equation, are not involved in the same processor. Data communication between adjacent processors is necessary after each iteration. In this paper, there is an imaginary buffer established between the two processors. After the electric field on the truncated of CPU is updated at each time step, $e_x$ obtained by interpolation method from coarse grid is transferred to the imaginary buffer for updating the magnetic field on the truncated of GPU in the next time. Similarly, after the magnetic on the truncated of GPU is updated at each time step, $H_z$ obtained by weighted summation method from dense grid is transferred to the imaginary buffer for updating the electric field on the truncated of the CPU in the next time.

3. Results

In this section, we apply the subgridding algorithm to simulate the air breakdown in the high-power microwave device. As shown in inset of Figure 6, the solution domain is an infinite wall with periodic PEC apertures, where the thickness is 0.3 mm, the length is 5 mm, and the width is 1.2 mm. A sinusoidal temporal profile plane wave polarized in the z-direction with a central frequency of 25 GHz and amplitude of 2 MV/m is launched at the TF/SF boundary and propagates toward the y-direction. The y-direction is truncated using the PMLs, and periodic boundary conditions are used in x- and z-direction. The initial electron density is assumed to be uniform (equal to $1\times {10}^{15}{ \mathrm{m}}^{-3}$) in the aperture region at standard atmospheric pressure. The computational domain, including the metallic aperture, is meshed by dense grids and sandwiched by two coarse mesh regions for free-space domains.

In Figure 6, we plot the evolvement of the plasma density at P1 for different GR. The electron density increases exponentially from its initial value of $1\times {10}^{15}{ \mathrm{m}}^{-3}$ to over $1\times {10}^{21}{ \mathrm{m}}^{-3}$ within the first 0.5 nanoseconds, indicating that the electron density is increasing massively from its initial value and then maintains s steady state. For different GRs, the results are in good agreement.

We recorded the $E_z$ component at points P1 (the center of aperture) and P2 (the distance between P1 and P2 is 2.81 mm). Figure 7 shows the temporal waveform of the electric fields recorded at P1 and P2 for GR = 3, 5, 7 and 9. The results are consistent with the results in [13]. From Figure 7, one can see that the electric field at P1 increases initially due to the field confinement of the metallic aperture. While the electric field magnitude rises above the breakdown threshold, the electric field magnitude dramatically decreases. The temporal profile of the electric field at P2 also demonstrates a similar result after the air breakdown; that is, the existence of plasma hinders the field transmission. Moreover, relative errors in dB at P2 are also shown in Figure 7. For different GR, there are maximum relative errors when the air starts ionizing. After 0.5 ns, the relative error is at much lower values.

To acquire a deep understanding of the physical process and compare the results between the dense grid and subgrid, the spatial and temporal evolution of the effective field $E_{eff}$ is presented in Figure 8, where (a), (b), (c), (d), and (e) represent the field distributions at 0.01 ns, 0.36 ns, 0.42 ns, 0.45 ns, and 0.1 ns, respectively. At 0.01 ns, due to the edge effect, the electric field at the aperture edge begins to increase, leading to the ionization of air. Within a short period, electron density increases dramatically. Then, the plasma formed by ionization will further cause electric field enhancement around the plasma. This process is shown in Figure 8b,c. In Figure 8d, the center of the aperture reaches the maximum value, the air in the center of the aperture also starts ionizing. After ionization of the entire aperture is completed, the plasma hinders the propagation of electromagnetic waves, as shown in Figure 8e.

To validate our subgridding FDTD method and benchmark the speed of the CPU/GPU heterogeneous computing technique, the proposed method was compared with a conventional FDTD CPU where serial code was running. The CPU used here was an Intel Core i9-10900K @ 3.70 GHz. The GPU used was a NVIDIA GeForce RTX 3090. The required memory and simulation time for different GRs are listed in

Table 1. Memory ratio and speedup in conventional FDTD (CPU) and subgridding FDTD (CPU-GPU) for different grid ratio.

Grid Ratio	3	5	7	9
Memory Ratio	1.0033	1.5078	2.3001	3.0641
Speedup	7.6316	19.4783	29.3629	35.0090

. Compared with conventional FDTD, the subgridding FDTD accelerated by the heterogeneous computing technique shows noticeable computation time and memory savings.

4. Conclusions

In this paper, A 3-D Maxwell–plasma model based on a subgridding FDTD algorithm has been developed and accelerated with the CPU/GPU heterogeneous computing technique. The results demonstrate that the subgridding FDTD can use less memory and requires a shorter time to obtain the same precision as the conventional FDTD method. For the subgridding FDTD, a smaller grid can get more accurate results. According to the CFL condition, the timestep should be pretty small, leading to a long simulation time. We used the CPU/GPU heterogeneous computing technique to accelerate this algorithm, and the results show that the proposed heterogeneous computing technique can effectively improve the simulation efficiency. The algorithm will be helpful for multiphysics and multiscale problems.