Towards Ab-Initio Simulations of Crystalline Defects at the Exascale Using Spectral Quadrature Density Functional Theory

Abstract

Defects in crystalline solids play a crucial role in determining properties of materials at the nano, meso- and macroscales, such as the coalescence of vacancies at the nanoscale to form voids and prismatic dislocation loops or diffusion and segregation of solutes to nucleate precipitates, phase transitions in magnetic materials via disorder and doping. First principles Density Functional Theory (DFT) simulations can provide a detailed understanding of these phenomena. However, the number of atoms needed to correctly simulate these systems is often beyond the reach of many widely used DFT codes. The aim of this article is to discuss recent advances in first principles modeling of crystal defects using the spectral quadrature method. The spectral quadrature method is linear scaling with respect to the number of atoms, permits spatial coarse-graining, and is capable of simulating non-periodic systems embedded in a bulk environment, which allows the application of appropriate boundary conditions for simulations of crystalline defects. In this article, we discuss the state-of-the-art in ab-initio modeling of large metallic systems of the order of several thousand atoms that are suitable for utilizing exascale computing resourses.

1. Introduction

Crystal defects, often present in small concentrations, are crucial in determining macroscopic properties of materials. Vacancies are fundamental to creep, spall and radiation aging, even when they are present in parts per million. Solutes, present in parts per hundred, are responsible for strengthening by interacting with the motion of dislocations. Solutes can also cluster to nucleate a precipitate. Dislocations are the primary carriers of plasticity in metals even when their density is as small as ${10}^{-8}$ per atomic row. Defects not only influence structural properties but also affect electronic and magnetic properties as well. For example, p-type and n-type semiconductors are primarily achieved by dopants. Doping and site mixing by magnetic ions in magnetic compounds can bring about magnetic phase transitions. Defects affect the macroscopic properties of crystals. This is because they simultaneously couple the chemical effects of the defect core, the discrete effects of the lattice and the long range effects of the elastic fields. Thus, to understand defects at physically relevant concentrations, we need to simultaneously study the details near the core and the long range elastic fields.

Density Functional Theory (DFT), developed by Hohenberg, Kohn and Sham [1,2], is one of the most successful first principles methods for predicting, understanding and developing insights into a wide range of material behavior. The phenomenal success and popularity of DFT is because of its excellent predictive power with high accuracy to cost ratio compared to other theories that solve for the electronic structure. In spite of this, the efficient solution of Kohn–Sham equations is computationally daunting, as the computational complexity of DFT calculations scale cubically with the number of atoms N (i.e., $\mathcal{O}\left(N^3\right)$). This restricts the range of physical systems that can be investigated. A number of methods have been proposed that reduce the prefactor by subspace projection [3,4,5,6]. Other methods focus on overcoming the cubic scaling bottleneck which can scale sub-quadratic [7] linear [8,9] and sub-linear [10] with the number of atoms. Methods that solve for orbitals achieve linear scaling by localizing the orbitals in real-space. However this approach only works for insulating or semiconducting systems. Other linear scaling methods do not solve the orbitals, and instead are based on the exponential decay of the density matrix and expand the density matrix using polynomials followed by truncation of off-diagonal components. The primary issue with this type of approach is the complicated interprocessor communication arising from distributed matrix–matrix products with changing sparsity patterns.

The Spectral Quadrature method is an alternative linear scaling formulation [8,10,11,12,13,14,15] suited for first-principles simulations of metallic as well as insulating and semiconducting systems. The key idea of this formulation is to write the electron density and energies as integrals over the spectrum of the linearized Hamiltonian, and then approximate these integrals by quadrature rules. The order of the quadrature is independent of system size, thereby allowing the evaluation of the electron density and energy with $\mathcal{O}\left(1\right)$ effort at each spatial point. The number of spatial points scale linearly with system size, and therefore the method scales linearly. This method has other attractive features as well. First, it allows the application of non-zero Dirichlet boundary conditions on the electronic fields [15]. This is specifically attractive for studying crystalline defects, where periodic boundary conditions can introduce artificial interactions. Second, the local nature of the spectral quadrature calculation allows variable resolution and coarse graining [8,10,14]. Specifically, fine resolution is maintained where necessary and sub-grid sampling is employed in regions of uniform deformation. See Figure 1 for different modeling schemes used for studying defects.

The spectral quadrature method and the coarse grained approximation have been applied to the study of crystal defects [10,14,15], and high temperature simulations of materials [13,19,20]. In this paper, we present an overview of the spectral quadrature formulation and discuss results on defects in magnesium.

2. Spectral Quadrature

In Kohn–Sham Density Functional Theory, the ground state energy (${\mathcal{E}}_0$) of a system with atomic positions $\mathbf{R}$ is given by minimizing the Kohn–Sham energy functional $\mathcal{E}$ over the set of orbitals $\left\{{\psi }_n\right\}$.

$${\mathcal{E}}_0=\underset{\left\{{\psi }_n\right\}}{inf}\mathcal{E}(\left\{{\psi }_n\right\},\mathbf{R}).$$

(1)

The orbitals $\left\{{\psi }_n\right\}$ are orthonormal, which give rise to the typical cubic scaling of Kohn–Sham DFT. As a consequence of their orthonormality, the orbitals are long ranged, oscillatory and do not decay in metallic systems. Several linear scaling approaches have been developed by localizing or truncating the orbitals in real space. Though these approaches are limited to insulating and semiconducting systems, they do not work for metallic systems i.e., absence of a band-gap. An alternate route is to work with the density operator formulation of DFT. The density operator ($\gamma$) is

$$\gamma =\sum _{n}g\left({ϵ}_n\right){\psi }_n\left(\mathbf{x}\right){\psi }_n\left({\mathbf{x}}^{\prime }\right),$$

(2)

where $g\left({ϵ}_n\right)$ are the occupancies of the $n^{th}$ orbital with energy ${ϵ}_n$.

The Kohn–Sham energy functional can be expressed in terms of the density operator, and the ground state energy is then given by minimizing this functional over the density operator.

$${\mathcal{E}}_0=\underset{\gamma }{inf}\mathcal{E}(\gamma ,\mathbf{R}).$$

(3)

The diagonal of the density operator is the electron density (i.e., $\rho =diag\left(\gamma \right)$).

The solution to Equation (3) is given by the following fixed-point problem:

$$\gamma =2{\left(1+exp\left(\frac{\mathcal{H}-\mu \mathcal{I}}{\sigma }\right)\right)}^{-1},$$

(4)

where $\mathcal{H}$ is the Kohn–Sham Hamiltonian, $\mu$ is the Fermi level, and $\sigma =K_{B}T$ is the electronic smearing. The right side of Equation (3) can be expanded as a polynomial of $\mathbf{H}$, the discrete version of $\mathcal{H}$. Linear scaling methods based on the polynomial expansion of $\mathbf{H}$ are known as Fermi Operator Expansion (FOE), make use of the sparsity, off-diagonal decay of the density operator, and ignore the entries of matrix-matrix products which are lower than a certain tolerance. In large-scale simulations using the FOE method, $\mathbf{H}$ is distributed across several computational cores, and the global matrix-matrix multiplications with changing sparsity patterns can give rise to complicated parallel communication.

The Spectral Quadrature method [8,10,11,12,13,14,15] is an alternate linear scaling formulation of DFT where the essential ground state quantities are expressed as integrals over the spectrum of the Hamiltonian.

$$I\left[f\right]={\int }_{\sigma \left(\mathcal{H}\right)}f\left(\lambda \right)\mathrm{d}{\mu }_{\eta ,\eta }\left(\lambda \right),$$

(5)

where $\sigma \left(\mathcal{H}\right)$ is the spectrum of $\mathcal{H}$, and ${\mu }_{\eta ,\eta }$ is the spectral measure of $\mathcal{H}$ contracted with $\eta$. The above integral (Equation (5)) can be approximated using quadratures.

$$I\left[f\right]\approx \sum _{k=1}^K{w}_k^{\eta }f\left({\lambda }_k^{\eta }\right),$$

(6)

where K is the order of the quadrature $w_k^{\eta }$ are the spectral weights and ${\lambda }_k^{\eta }$ are the spectral nodes. The quadrature order is independent of the system size, hence each evaluation of Equation (6) is $\mathcal{O}\left(1\right)$. The summation (Equation (6)) is evaluated at every discrete point in real space and the number of such discrete points scale linearly with the number of atoms N, and therefore the scaling of evaluating the ground state is linear.

In spectral Gauss quadrature, the weights ${\left\{w_k^{\eta }\right\}}_{k=1}^K$ are fixed apriori, and the spectral nodes ${\left\{{\lambda }_k^{}\eta \right\}}_{k=1}^K$ are treated as unknowns. The spectral weights and nodes at any grid point q can be calculated by the Lanczos iteration,

$$\begin{array}{ccc}\hfill b_{k+1}^q{v}_{k+1}^q& =& (\mathbf{H}-a_{k+1}^q)v_k^q-b_k^q{v}_{k-1}^q,k=0,1,\dots ,K-1\hfill \\ \hfill v_{-1}^q& =& 0,v_0^q={\eta }_q,b_0^q=1,\hfill \end{array}$$

(7)

where $a_{k+1}^q=(\mathcal{H}{v}_k^q,v_k^q)$, $k=0,1,\dots ,K-1$, and $b_k^q$ is computed such that $\left|\right|v_k^q\left|\right|=1,k=0,1,\dots ,K-1$. ${\left\{a_k^q\right\}}_{k=1}^K$ and ${\left\{b_k^q\right\}}_{k=1}^{K-1}$ are used to construct a tri-diagonal Jacobi matrix ${\widehat{J}}_K^q$, where $(a_1^q,a_2^q,\dots ,a_K^q)$ is the diagonal and $(b_1^q,b_2^q,\dots ,b_{K-1}^q)$ are the subdiagonal and superdiagonal entries. The nodes ${\left\{w_k^{{\eta }_q}\right\}}_{k=1}^K$ are calculated as the eigenvalues of ${\widehat{J}}_K^q$, and the weights ${\left\{{\lambda }_k^{{\eta }_q}\right\}}_{k=1}^K$ are the squares of the first elements of the normalized eigenvectors of ${\widehat{J}}_K^q$. The Lanczos iteration (Equation (7)) is the most computationally expensive part of the entire computation and can be accelerated by offloading onto Graphics Processing Units (GPUs).

After the spectral weights and spectral nodes are evaluated at a grid point q using the Lanczos scheme (Equation (7)), the electron density at the grid point is given by

$${\rho }^q\approx 2\sum _{k=1}^K{w}_k^{{\eta }_q}{\left(1+exp\left(\frac{{\lambda }_k^q-\mu }{\sigma }\right)\right)}^{-1}.$$

(8)

The electron density, and similarly other electronic fields, can be calculated in a linear scaling fashion using the spectral quadrature technique. The prefactor can be reduced by noting that the growth of vectors ${\left\{v_k^q\right\}}_{k=1}^K$ are restricted within a sphere of a finite radius—typically a few lattice spacings—around the grid point q. Therefore, the discrete Hamiltonian in Equation (7), can be spatially truncated beyond a certain distance. This insight allows us to employ local sparse algebra routines instead of distributed sparse algebra, decreases computation time, and storage.

Spatial Coarse Graining

The Spectral Quadrature Method can be used for large scale first principles simulations of materials. Typical calculations with the Spectral quadrature method involve several hundred to tens of thousands of atoms. Further controlled approximations can be introduced which can push the number of atoms even higher using the idea of spatial coarse-graining [8,10,14,21]. This is specifically advantageous for simulating extended defects such as dislocations. To understand heuristics involved in the approximations, first consider an isolated defect. The electronic fields and the atomic displacements are complex in the vicinity of the defect. Moving far away from the defect, the atomic displacements decay in either a polynomial of logarithmic manner, and the electronic fields far away from the defect are almost locally periodic. In the coarse grained formulation, the nature of the decay of the elastic and the electronic fields are exploited in the following manner. First, the position of all atoms is approximated by tracking only a subset ${\mathcal{P}}_{RA}$ of Representative Atoms (RAs) which are fully dense in the vicinity of the defect core and is increasingly sparse with increase in distance from the defect. The positions of atoms which are not in ${\mathcal{P}}_{RA}$ are evaluated by linear interpolation, which provides a coarse grained representation of the atomistic fields. Next, for the electronic fields, the discrete Hamiltonian is defined uniformly on a spatial finite difference mesh ${\mathcal{P}}_f$. However, the electronic fields are not evaluated at all points in ${\mathcal{P}}_f$. Instead, a different representation is employed based on the observation that sufficiently far away from the defect, the atom positions are locally periodic, where the periodicity is related to the local strain, and the electronic fields are almost locally periodic. Due to this, the electronic fields can be written as the sum of a corrector and a locally periodic predictor in the following way:

$${\rho }^q={\rho }_0^q+{\rho }_c^q,$$

(9)

where ${\rho }_0^q$ is the piecewise periodic predictor and ${\rho }_c^q$ is the corrector at a mesh point $q\in {\mathcal{P}}_f$. The predictor can be calculated by using a periodic DFT calculation with the lattice parameters subjected to local strain. To calculate the corrector, the electron density is first calculated at a select few points $p\in {\mathcal{P}}_c$ using Spectral quadrature, and the difference between the electron density and predictor is interpolated onto the points in ${\mathcal{P}}_f$:

$${\rho }_c^q=\sum _{p\in {\mathcal{P}}_c}{\Gamma }_p^q({\rho }^p-{\rho }_0^p),$$

(10)

where the ${\mathcal{P}}_c$ is a subset of ${\mathcal{P}}_f$. Once the electronic fields are calculated, the total energy can be computed using cluster summation technique [22]. Figure 2a–c shows the features of defects and the associated atomic and electronic mesh for a one dimensional system.

Note that, because the electron density is calculated only on a select few points in ${\mathcal{P}}_c$, the scaling is sub-linear with respect to the total number of atoms in the domain. Figure 2d shows the representative plot of computation time versus number of atoms for cubic, linear and sub-linear scaling methods.

In the coarse-grained formulation, a key computational expense comes from repeated on-the-fly calculation of the predictor electronic fields for every local strain field. This significant overhead can be reduced by employing a neural-network which maps the local strain to the electronic fields. We refer the reader to the recent work by Teh, Ghosh and Bhattacharya [23] for constructing such a map using machine learning.

3. Discussion

3.1. Bulk Properties of Elements

The Spectral Quadrature Method was used to calculate the bulk properties of materials. Higher order finite differences were used to represent the discrete Hamiltonian operator. The finite difference mesh spacing and tolerances were chosen such that the energies and forces are converged to within chemical accuracies.

Table 1. Bulk properties of elements.

Element	Crystal Structure	Method	Lattice Constant (a.u.)	Bulk Modulus (GPa)
Li	BCC	$O\left(N\right)$ SQ [11]	6.87	10.0
		$O\left(N^3\right)$ DFT [24]	6.77	14.0
		Expt. [24]	6.77	13.3
Na	BCC	$O\left(N\right)$ SQ [11]	8.01	5.0
		$O\left(N^3\right)$ DFT [24]	8.21	7.1
		Expt. [24]	8.21	7.3
Mg	HCP	$O\left(N\right)$ SQ [10]	$a=$ 5.866, $c/a=$ 1.626	38.75
		$O\left(N\right)$ SQ [15]	$a=$ 6.043, $c/a=$ 1.629	38.50
		$O\left(N^3\right)$ DFT [25]	$a=$ 5.877, $c/a=$ 1.624	38.40
		Expt. [26,27]	$a=$ 6.066, $c/a=$ 1.623	35.40

shows the values of equilibrium lattice constant and bulk modulus for Li, Na and Mg. Overall, the values calculated using the $\mathcal{O}\left(N\right)$ spectral quadrature DFT are in excellent agreement with the values calculated using $\mathcal{O}\left(N^3\right)$ DFT, and experiments.

3.2. Defects in Magnesium

Defects in crystalline materials give rise to several interesting phenomena such as creep, spall, plasticity, radiation aging, phase transitions. Defects are present in very small concentrations which make their accurate study challenging. Defect can also interact with other defects in crystalline solids. Vacancies can coalesce to nucleate voids, solutes cluster to nucleate a precipitate, solutes bind to vacancies for diffusion, solutes interact with dislocations for strengthening.

We consider divacancy, solute–vacancy and solute–solute pairs in magnesium lattice. Figure 3 shows the six nearest neighbor positions on a HCP lattice where the first defect is placed at the position marked 0 and the other defect occupies positions marked 1 through 6. These denote the first six nearest neighbor configurations in increasing order of their distances from the defect at position 0. Let ${\mathcal{E}}_{vv}$, ${\mathcal{E}}_{sv}$ and ${\mathcal{E}}_{ss}$ denote the energies of divacancy, solute–vacancy and solute–solute pairs, respectively in a magnesium supercell with M lattice sites. The formation energies of the divacancy pair is:

$$E_{vv}^f={\mathcal{E}}_{vv}-(M-2){\mathcal{E}}_{perfect},$$

(11)

where ${\mathcal{E}}_{perfect}$ is the energy of a magnesium atom in the perfect crystalline state. The formation energy of the solute–vacancy pair is:

$$E_{sv}^f={\mathcal{E}}_{sv}-(M-2){\mathcal{E}}_{perfect}-{\mathcal{E}}_{solute},$$

(12)

where ${\mathcal{E}}_{solute}$ is the energy of a solute atom in in the perfect crystalline state. The formation energy of the solute–solute pair is:

$$E_{ss}^f={\mathcal{E}}_{ss}-(M-2){\mathcal{E}}_{perfect}-2{\mathcal{E}}_{solute}.$$

(13)

A negative formation energy implies that the defect formation is energetically favorable.

The binding energy of defect pair can be calculated as the difference between the sum of formation energies of the individual isolated defects and the defect pair.

$$\begin{array}{ccc}\hfill E_{vv}^b& =& 2{\mathcal{E}}_v^f-{\mathcal{E}}_{vv}^f,\hfill \end{array}$$

(14)

$$\begin{array}{ccc}\hfill E_{sv}^b& =& {\mathcal{E}}_s^f+{\mathcal{E}}_v^f-{\mathcal{E}}_{sv}^f,\hfill \end{array}$$

(15)

$$\begin{array}{ccc}\hfill E_{ss}^b& =& 2{\mathcal{E}}_s^f-{\mathcal{E}}_{ss}^f,\hfill \end{array}$$

(16)

where ${\mathcal{E}}_v^f$ and ${\mathcal{E}}_s^f$ are the formation energies of vacancies and solutes respectively. A positive binding energy implies that the defect pair is more stable than the individual isolated defects.

In Figure 4, the top-panel shows the formation energies of (a) divacancy, (b) Al solute–vacancy and (c) Al solute–solute pairs, respectively with increasing cell size from 64 to 1152 atoms. From these plots we see that the formation energy of defect pairs strongly depend on the cell size. Furthermore, the solute–solute formation energy calculated at cell sizes less than 200 atoms is negative but is positive at larger cell sizes. The bottom panel of Figure 4 shows the contours of the difference in electron density of the defect pairs in the second nearest neighbor configuration. We see the depletion of charge around vacancies and an accumulation of charge around the solute atoms.

Table 2. Formation energies of vacancy and Al solute in Mg lattice in eV. Formation and binding energies of divacancies, Al solute–vacancy and Al solute–solute pairs in Mg in eV. First six nearest neighbors are considered for calculation of defect pair binding energies as shown in Figure 3. The cell size used for all calculations is more than 1100 atoms.

Isolated Defects
	Formation Energy (eV)
Vacancy	0.846
Al solute	25.756
Defect Pairs
	Nearest Neighbor	Formation Energy (eV) This Work	Binding Energy (eV) Ref. [15]
Divacancy	1	1.565	0.127
	2	1.596	0.195
	3	1.627	0.064
	4	1.659	0.033
	5	1.659	0.033
	6	1.659	0.033
solute-vacancy	1	25.956	0.254
	2	26.583	0.195
	3	26.583	0.206
	4	26.722	0.125
	5	26.722	0.125
	6	26.722	0.125
solute-solute	1	1.066	0.238
	2	1.081	0.223
	3	1.084	0.219
	4	1.112	0.191
	5	1.134	0.169
	6	1.166	0.138

shows the formation and binding energies of isolated defects and defect pairs calculated using the spectral quadrature method. The formation energy of a monovacancy and Al solute is positive. Further the formation energies of divacancy, solute–vacancy and solute–solute pairs are also positive. Interestingly, the formation energy increases from ∼1 eV to ∼26 eV when the system is changed from a divacancy to an Al solute–vacancy pair, but decreases to ∼1 eV for solute–solute pair. Thus, the formation energy follows a non-linear trend with increasing solute number. The equilibrium concentration of spontaneously formed vacancies and divacancies depend on the formation energy and temperature as $C=exp(-E^f/k_{B}T)$. As the formation energies of divacancies are greater than monovacancies, the spontaneous formation of monovacancies is favored over the spontaneous formation of divacancies. Additionally, the binding energies of divacancies are positive, implying two monovacancies can coalesce to form a divacancy. Again from Table 2, the binding energies of solute–vacancy and solute–solute pairs are also positive. The solute–vacancy binding energy is also greater than the divacancy binding energy, which implies that solute and vacancy can strongly bind, which aids the self diffusion of solute atoms. As the solute–solute binding energy is also large, these solutes atoms can bind to each other, form clusters and nucleate precipitates [15].

The Spectral Quadrature Method can be used to simulate even larger systems, containing several tens of thousands of atoms. Figure 5 shows the electron density contours around a prismatic dislocation loop in Magnesium calculated using 24,539 atoms, and the prismatic loop consists of a cluster of 37 vacancies. The cell sizes used in these calculations are beyond the scope of most widely popular DFT codes. Calculations with coarse-grained spectral quadrature reveal the preference towards forming large stable vacancy clusters and preference towards forming large prismatic dislocation loops by clustering of vacancies, as observed in nanoporous Mg alloys and experiments [14]. Further optimizing the Lanczos iteration into Graphics Processing Units (GPUs) and utilizing exascale resources (e.g., Frontier) will significantly increase the size of systems to several hundred thousand atoms.

We finally close by commenting on the applicability of the above method to problems in the mechanical behavior of materials. It is of interest to calculate the influence of defects on the mechanical moduli of a crystal. Simulating processes involving material flow [28] and additive manufacturing [29] require accurate estimation of effective moduli of materials under evolving defects such as voids and precipitate growth. Micromechanics approaches dating back to the work of Eshelby [30], Hashin and Shtrikman [31] provide upper and lower bounds on the moduli. Traditional DFT methods have been used to calculate the moduli under high defect concentrations [32]. Therefore, it will be interesting to use the spectral quadrature method on exascale computing resources to calculate the influence of voids and precipitates at high or realistic concentrations from ab-initio.

4. Conclusions

In this article, we discuss recent advances in first principles simulations of bulk materials and crystalline defects using the Spectral Quadrature Computational method. This framework is capable of simulating non-periodic systems embedded in a bulk environment, which allows the application of correct boundary conditions for simulations of crystalline defects. Furthermore, due to the local representation of the electronic fields, this method is also amenable to spatial coarse graining. We have presented studies of defect clusters in magnesium using cell sizes ranging from a few tens to several tens of thousands of atoms. These calculations are typically beyond the reach of the most widely used DFT codes. Due to the local nature of all calculations, the method is amenable to spatial coarse-graining which can be further accelerated by utilizing machine learning models. Furthermore, the local computations can be transferred onto Graphics Processing Units (GPUs), which significantly reduces the solution times. These features make the formulation suitable for efficiently utilizing the exascale supercomputing frameworks.