Accurate Sampling with Noisy Forces from Approximate Computing

Abstract

In scientific computing, the acceleration of atomistic computer simulations by means of custom hardware is finding ever-growing application. A major limitation, however, is that the high efficiency in terms of performance and low power consumption entails the massive usage of low precision computing units. Here, based on the approximate computing paradigm, we present an algorithmic method to compensate for numerical inaccuracies due to low accuracy arithmetic operations rigorously, yet still obtaining exact expectation values using a properly modified Langevin-type equation.

1. Introduction

Molecular dynamics (MD) is a very powerful and widely used technique to study thermodynamic equilibrium properties, as well as the real-time dynamics of complex systems made up of interacting atoms [1]. This is done by numerically solving Newton’s equations of motion in a time-discretized fashion via computing the nuclear forces of all atoms at every time step [2]. Computing these forces by analytically differentiating the interatomic potential with respect to the nuclear coordinates is computationally rather expensive, which is particularly true for electronic structure-based ab-initio MD simulations [3,4,5,6].

For a long time, newly developed microchips have become faster and more efficient due to new manufacturing processes and shrinking transistor sizes. However, this development is slowly coming to an end as scaling down the structures of silicon-based chips becomes more and more difficult. The focus therefore shifts towards making efficient use of the available technology. Hence, besides algorithmic developments [7,8,9,10,11,12,13,14], there have been numerous custom computing efforts in this area to increase the efficiency of MD simulations by means of hardware acceleration, which we take up in this work. Examples of the latter are MD implementations on graphics processing units (GPUs) [15,16,17,18,19,20,21], field-programmable gate arrays (FPGAs) [22,23], and application-specific integrated circuits (ASICs) [24,25]. While the use of GPUs for scientific applications is relatively widespread [26,27,28], the use of ASICs [29,30,31,32] and FPGAs is less common [33,34,35,36,37,38], but has gained attention over the last few years. In general, to maximize the computational power for a given silicon area, or equivalently minimize the power-consumption per arithmetic operation, more and more computing units are replaced with lower precision units. This trend is mostly driven by the market considerations of the gaming and artificial intelligence industries, which are the target customers of hardware accelerators and naturally do not absolutely rely on full computing accuracy.

In the approach presented in this paper, we mimic in software how it is possible to make effective use of low accuracy special-purpose hardware for general-purpose scientific computing by leveraging the approximate computing (AC) paradigm [39,40]. The general research goal of AC is to devise and explore ingenious techniques to relax the exactness of a calculation to facilitate the design of more powerful and/or more efficient computer systems. However, in scientific computing, where the exactness of all computed results is of paramount importance, attenuating accuracy requirements is not an option. Yet, assuming that the inaccuracies within the nuclear forces due to the usage of low precision arithmetic operations can be approximately considered as white noise, we will demonstrate that it is nevertheless possible to compensate for such numerical errors rigorously and still obtain exact expectation values, as obtained by ensemble averages of a properly modified Langevin equation.

The remainder of the paper is organized as follows. In Section 2, we revisit the basic principles of AC before introducing our modified Langevin equation in Section 3. Thereafter, in Section 4, we describe the computational details of our computational experiments. Our results are presented and discussed in Section 5 before concluding the paper in Section 6.

2. Approximate Computing

A basic method of approximation and a key requirement for efficient use of processing hardware is the use of adequate data widths in computationally intensive kernels. While in many scientific applications, the use of the double-precision floating-point is most common, this precision is not always required. For example, iterative methods can exhibit resilience against low precision arithmetic, as has been shown for the computation of inverse matrix roots [41] and for solving systems of linear equations [39,42,43,44]. Mainly driven by the growing popularity of artificial neural networks [45], we can observe growing support of low precision data types in hardware accelerators. In fact, recent GPUs targeting the data center have started supporting half-precision as well, nearly doubling the peak performance compared to single-precision and quadrupling it compared to double-precision arithmetics [46]. However, due to the low number of exponent bits, half-precision only provides a very limited dynamic range. In contrast, bfloat16 provides the same dynamic range as single-precision and just reduces precision. It is currently supported by Google’s Tensor Processing Units (TPU) [47], and support has been announced for future Intel Xeon processors [48] and Intel AgileX FPGAs. A list of commonly used data types, together with the corresponding number of bits used to store the exponent and the mantissa, are shown in

Table 1. Bitwidth of common floating-point formats.

Type	Sign	Exponent	Mantissa
IEEE 754 quadruple-precision	1	15	112
IEEE 754 double-precision	1	11	52
IEEE 754 single-precision	1	8	23
IEEE 754 half-precision	1	5	10
Bfloat16(truncated IEEE single-precision)	1	8	7

beside the double-precision de facto standard.

Yet, programmable hardware such as FPGAs, as a platform for custom-built accelerator designs [49,50,51], can make effective use of all of these, but also entirely custom number formats. Developers can specify the number of exponent and mantissa bits and trade-off precision against the amount of memory blocks required to store values and the number of logic elements required to perform arithmetic operations on them.

In addition to floating-point formats, also fixed-point representations can be used. Here, all numbers are stored as integers of a fixed size with a predefined scaling factor. Calculations are thereby performed using integer arithmetic. On CPUs and GPUs, only certain models can perform integer operations with a peak performance similar to that of floating-point arithmetic, depending on the capabilities of the vector units/stream processors. Nevertheless, FPGAs typically can perform integer operations with performance similar to or even higher than that of the floating-point. Due to the high flexibility of FPGAs with respect to different data formats and the possible use of entirely custom data types, we see them as the main target technology for our work. For this reason, we consider both floating-point and fixed-point arithmetic in the following.

3. Methodology

The error introduced by low precision floating-point or fixed-point computations can in general be modeled as white noise if unbiased rounding techniques are used in all arithmetic operations. A widely employed rounding technique is round half to even, which does not introduce a systematic bias and is used by default in IEEE 754 floating-point arithmetic [52]. In the following, we assume the usage of such a rounding technique also for fixed-point arithmetic, leading to only an unbiased error within the computed interatomic forces.

To demonstrate the concept of approximate computing, we introduce white noise to the interatomic forces that are computed while running the MD simulation. In this section, we describe in detail how we introduce the noise to mimic in software the behavior that would be observed when running the MD on the actual FPGA or GPU hardware with reduced numerical precision. We classify the computational errors into two types: fixed-point errors and floating-point errors. Assuming that ${\mathbf{F}}_I$ are the exact and ${\mathbf{F}}_I^N$ the noisy forces from an MD simulation with low precision on an FPGA for instance, fixed-point errors can by modeled by:

$${\mathbf{F}}_I^N=\left(\begin{array}{c}{\mathrm{F}}_I^x\\ {\mathrm{F}}_I^y\\ {\mathrm{F}}_I^z\end{array}\right)+\left(\begin{array}{c}{c}_1\times {10}^{-\beta }\\ c_2\times {10}^{-\beta }\\ c_3\times {10}^{-\beta }\end{array}\right),$$

(1)

whereas floating-point errors are described by:

$${\mathbf{F}}_I^N=\left(\begin{array}{c}{\mathrm{F}}_I^x\times {10}^{-{\alpha }_1}\\ {\mathrm{F}}_I^y\times {10}^{-{\alpha }_2}\\ {\mathrm{F}}_I^z\times {10}^{-{\alpha }_3}\end{array}\right)+\left(\begin{array}{c}{c}_1\times {10}^{-({\alpha }_1+\beta )}\\ c_2\times {10}^{-({\alpha }_2+\beta )}\\ c_3\times {10}^{-({\alpha }_3+\beta )}\end{array}\right).$$

(2)

Therein, $c_1$, $c_2$, and $c_3$ are random values chosen in the range [−0.5, 0.5], whereas ${\mathrm{F}}_I^x$, ${\mathrm{F}}_I^y$, and ${\mathrm{F}}_I^x$ are the individual force components of ${\mathbf{F}}_I$, respectively. The floating-point scaling factor is denoted as $\alpha$ and the magnitude of the applied noise by $\beta$.

To correct the errors introduced by numerical noise rigorously, we employ a modified Langevin equation. In particular, we model the force as obtained by a low precision computation on a GPU or FPGA-based accelerator as:

$${\mathbf{F}}_I^N={\mathbf{F}}_I+{\Xi }_I^N,$$

(3)

where ${\Xi }_I^N$ is an additive white noise for which:

$$〈{\mathbf{F}}_I\left(0\right){\Xi }_I^N\left(t\right)〉\cong 0$$

(4)

holds. Throughout, $\langle \cdots \rangle$ denotes Boltzmann-weighted ensemble averages as obtained by the partition function $Z=\mathrm{Tr}exp(-E/k_{B}T)$, where E is the potential energy, $k_B$ the so-called Boltzmann constant, and T the temperature. Given that ${\Xi }_I^N$ is unbiased, which in our case is true by its very definition, it is nevertheless possible to sample the Boltzmann distribution accurately by means of a Langevin-type equation [53,54,55], which in its general form reads as:

$$M_I{\ddot{\mathbf{R}}}_I={\mathbf{F}}_I+{\Xi }_I^N-{\gamma }_N{M}_I{\dot{\mathbf{R}}}_I,$$

(5)

where ${\dot{\mathbf{R}}}_I$ are the nuclear coordinates (the dot denotes the time derivative), $M_I$ are the nuclear masses, and ${\gamma }_N$ is a damping coefficient, which is chosen to compensate for ${\Xi }_I^N$. The latter, in order to guarantee an accurate canonical sampling, has to obey the fluctuation-dissipation theorem:

$$〈{\Xi }_I^N\left(0\right){\Xi }_I^N\left(t\right)〉\cong 2{\gamma }_N{M}_I{k}_{B}T\delta \left(t\right).$$

(6)

Substituting Equation (3) into Equation (5) results in the desired modified Langevin equation:

$$M_I{\ddot{\mathbf{R}}}_I={\mathbf{F}}_I^N-{\gamma }_N{M}_I{\dot{\mathbf{R}}}_I,$$

(7)

which will be used throughout the remainder of this paper. This is to say that the noise, as originating from a low precision computation, can be thought of as the additive white noise of a damping coefficient ${\gamma }_N$, which satisfies the fluctuation-dissipation theorem of Equation (6). The specific value of ${\gamma }_N$ is determined in such a way so as to generate the correct average temperature, as measured by the equipartition theorem:

$$〈\frac{1}{2}{M}_I{\dot{\mathbf{R}}}_I〉=\frac{3}{2}{k}_{B}T.$$

(8)

4. Computational Details

To demonstrate our approach, we implemented it in the CP2K suite of programs [56,57]. More precisely, we conducted MD simulations of liquid silicon (Si) at 3000 K using the environment-dependent interatomic potential of Bazant et al. [58,59]. All simulations consisted of 1000 Si atoms in a 3D-periodic cubic box of length 27.155 Å. Using the algorithm of Ricci and Ciccotti [60], Equation (5) was integrated with a discretized time step of 1.0 fs with ${\gamma }_N=0.001$ fs${}^{-1}$.

Whereas the latter settings were used to compute our reference data, in total, six different cases of fixed-point and floating-point errors were investigated by varying the exponent $\beta$ between zero (huge noise) and three (tiny noise) that is, ranging from $1/1000$ of the physical force up to the same magnitude as the force. As already alluded to above, the additive white noise is compensated via Equation (7) by continuously adjusting the friction coefficient ${\gamma }_N$ using the adaptive Langevin technique of Leimkuhler and coworkers so as to satisfy the equipartition theorem of Equation (8) [61,62,63]. In this method, ${\gamma }_N$ is reinterpreted as a dynamical variable, defined by a negative feedback loop control law as in the Nosé–Hoover scheme [64,65]. The corresponding dynamical equation for ${\gamma }_N$ reads as:

$${\dot{\gamma }}_N=(2K-n{k}_{B}T)/\mathcal{Q},$$

(9)

where K is the kinetic energy, n is the number of degrees of freedom, and $\mathcal{Q}=k_{B}T{\tau }_{NH}^2$ is the Nosé–Hoover fictitious mass with time constant ${\tau }_{NH}$. Alternatively, ${\gamma }_N$ can be estimated by integrating the autocorrelation function of the additive white noise [66]. In

Table 2. Values for ${\gamma }_N^{fix}$ and ${\gamma }_N^{float}$ as a function of $\beta$.

$\mathit{\beta }$	${\mathit{\gamma }}_{\mathit{N}}^{\mathit{f}\mathit{i}\mathit{x}}$	${\mathit{\gamma }}_{\mathit{N}}^{\mathit{f}\mathit{l}\mathit{o}\mathit{a}\mathit{t}}$
0		0.00025
1	0.0004	0.000005
2	0.000009	0.000005
3	0.0000009

, the resulting values of ${\gamma }_N^{fix}$ for fixed-point and ${\gamma }_N^{float}$ for floating-point errors are reported as a function of $\beta$.

5. Results and Discussion

As can be directly deduced from Table 2, the smaller values of ${\gamma }_N$ for a given $\beta$ immediately suggested a higher noise resilience when using floating-point as compared to fixed-point numbers.

In Figure 1 and Figure 2, the Si-Si partial pair-correlation function $g\left(r\right)$, which describes how the particle-density varies as a function of distance from a reference particle (atoms, molecules, colloids, etc.), as computed using an optimal scheme for orthorhombic systems [67], is shown for different values of $\beta$. As can be seen, for both fixed-point and floating-point errors, the agreement with our reference calculation was nearly perfect up to the highest noise we investigated. As already anticipated earlier, the usage of floating-point errors was not only able to tolerate higher noise levels, but was also more accurate throughout.

To verify that the sampling was indeed canonical, in Figure 3, the actual kinetic energy distribution as obtained by our simulations using noisy forces is depicted and compared to the analytic Maxwell distribution. It was evident that if sampled long enough, not only the mean value, but also the distribution tails were in excellent agreement with the exact Maxwellian kinetic energy distribution, which demonstrated that we were indeed performing a canonical sampling.

To further assess the accuracy of the present method, we expanded the autocorrelation of the noisy forces, i.e.,

$$\begin{array}{c}〈{\mathbf{F}}_I^N\left(0\right){\mathbf{F}}_I^N\left(t\right)〉\hfill \end{array}$$

(10a)

$$\begin{array}{cc}=& 〈\left({\mathbf{F}}_I\left(0\right)+{\Xi }_I^N\left(0\right)\right)\left({\mathbf{F}}_I\left(t\right)+{\Xi }_I^N\left(t\right)\right)〉\hfill \end{array}$$

(10b)

$$\begin{array}{cc}=& 〈{\mathbf{F}}_I\left(0\right){\mathbf{F}}_I\left(t\right)〉+〈{\mathbf{F}}_I\left(0\right){\Xi }_I^N\left(t\right)〉\hfill \\ +& 〈{\mathbf{F}}_I\left(t\right){\Xi }_I^N\left(0\right)〉+〈{\Xi }_I^N\left(0\right){\Xi }_I^N\left(t\right)〉.\hfill \end{array}$$

(10c)

Since the cross-correlation terms between the exact force and the additive white noise were vanishing due to Equation (4), comparing the autocorrelation of the noisy forces $\langle {\mathbf{F}}_I^N\left(0\right){\mathbf{F}}_I^N\left(t\right)\rangle$ with the autocorrelation of the exact forces $\langle {\mathbf{F}}_I\left(0\right){\mathbf{F}}_I\left(t\right)\rangle$ permitted assessing the localization of the last term of Equation (10c). The fact that $\langle {\mathbf{F}}_I^N\left(0\right){\mathbf{F}}_I^N\left(t\right)\rangle$ was essentially identical to $\langle {\mathbf{F}}_I\left(0\right){\mathbf{F}}_I\left(t\right)\rangle$, as can be seen in Figure 4, implied that $\langle {\Xi }_I^N\left(0\right){\Xi }_I^N\left(t\right)\rangle$ was very close to a $\delta$-function as required by the fluctuation-dissipation theorem in order to ensure an accurate canonical sampling of the Boltzmann distribution. In other words, from this, it followed that our initial assumption underlying Equation (7), to model the noise due to a low precision calculation as an additive white noise channel, was justified.

6. Conclusions

We conclude by noting that the presented method was recently implemented in the universal force engine i-PI [68], which can be generally applied to all sorts of forces affected by stochastic noise such as those computed by GPUs or other hardware accelerators [15,16,17,18,19,20,21], and potentially even quantum computing devices [69,70,71,72]. The possibility to apply similar ideas to N-body simulations [73,74] and to combine them with further algorithmic approximations [75] is to be underlined and will be presented elsewhere.