The overall accuracy of a chemical transport simulation, roughly defined as the difference between the model output and the real chemical concentration fields, is the result of nonlinear interactions between errors coming from different sources:

The overall simulation accuracy can be considerably improved by data assimilation [3,4]. Thus, the overall simulation error also depends on the amount of information carried by external measurements, and by the quality of the data assimilation system used [5–14].

Data and and modeling errors are difficult to quantify. A desirable level of overall numerical accuracy is on the order of 1%; the overall simulation accuracy can be assessed aposteriori by comparing model results and measurements. The size of the numerical errors should be at least one or two orders of magnitude below the overall target.

The chemical kinetic mechanism is just one subsystem of a large chemical transport simulation. Data errors are associated with the initial conditions. Model errors are associated with the level of detail of the chemical mechanism, and with the accuracy of reaction rate coefficient values. Numerical errors are associated with the particular numerical integration algorithm employed and with the length of the time steps used.

The target level of relative accuracy (relative error tolerance) is 0.1%, i.e., about 3 accurate digits. (Better accuracy is of course possible, but may demand longer compute times without improving the overall simulation accuracy.) The final solution accuracy is determined by the order of accuracy of the numerical algorithm, and by the sequence of step sizes (the temporal grid). Though both the time step and the order can be adjusted dynamically to achieve the desired accuracy, the most common error control mechanism is adjusting the step size.

Due to operator splitting [1], at the beginning of each integration interval the chemical system goes through a transient phase. Step sizes need to be small to resolve the transient, and need to quickly increase size after that. Therefore, numerical methods of choice are able to quickly adjust time steps (a property of one-step integration algorithms). A robust step size adaptivity mechanism is also needed. As pointed out in [2], solvers targeting large relative error levels are likely to work outside the asymptotic error regime for which they have been designed.

Different chemical species participating in atmospheric chemical kinetics have widely different life times. Specifically, different species evolve on different time scales, from milliseconds (e.g., for radicals such as OH) to years (e.g., for CH₄). The resulting system of ordinary differential equations is stiff [15], and special care needs to be exercised in the choice of the numerical integration scheme.

Due to numerical stability considerations, explicit time integration methods cannot use time steps that are much larger than the fastest time scale in the system. Roughly speaking, the current solution is influenced by the approximation error made during the previous step, multiplied by the ratio of the step size over the fastest dynamic time scale. If this ratio is large the errors accumulate extremely quickly and the solution becomes unusable (numerical instability).

The time stepping methods used to solve atmospheric chemical kinetics should be unconditionally stable, i.e., stable for any choice of the step size. A desirable property is L-stability [15], which implies that the scheme is stable for any eigenvalues of the Jacobian (any dynamics) and any step size, and that very high frequencies and very fast transients are completely damped out. General methods with this stability properties are necessarily implicit. Thus, at each step the solution is obtained by solving a nonlinear system of equations.

The solution of a chemical kinetic system has several intrinsic properties. The total mass and total electric charge are preserved during the system evolution, and the concentrations remain positive at all times. It is desirable that the numerical solution preserves such properties as well [16].

The total mass and the total charge are linear invariants of the system, i.e., they can be written as M = ∑ i = 1 d m i y i and Q = ∑ i = 1 d q i y i where d is the number of chemical species. Most of the general purpose time integration algorithms (Runge Kutta, linear multistep, and extrapolation methods) preserve linear invariants within roundoff error. Thus mass and charge preservation are almost automatic.

The preservation of positivity is more difficult to achieve. Methods that preserve positivity unconditionally (for any step size) are at most of order one [17]. For a typical integration method (implicit or explicit) the preservation of positivity restricts the time step to a small multiple of the fastest dynamic time scale; the step restriction due to positivity is as severe as that due to numerical stability for explicit methods. A simple solution to preserve positivity is clipping, where small negative concentrations are set to zero at each solution step. Clipping has the disadvantage that it consistently adds artificial mass to the system. (The total mass is no longer preserved within the roundoff error, but only within the truncation error). A more involved approach is positive projection [18] where the solution is computed at each step, and if some concentrations are negative, a projection onto the non-negative simplex is performed such as to conserve mass, and to preserve the accuracy of the original solution. Positive projection overcomes the order one barrier [17], but can add a significant computational overhead if the projection algorithm is called many times. A computationally lighter approach is offered by methods that favor positivity [19].

Note that the preservation of positivity is important only in those situations where negative concentrations render the ODE dynamics unstable. For many chemical mechanisms used in practice, small negative concentrations do not result in instability [18], thus the enforcement of positivity during integration is not a necessity and the small negative values can be removed in a post-processing stage.

Since the solution of chemical kinetics takes up an important fraction of the total compute cycles in a chemical transport simulation, special care needs to be paid to computational efficiency Roughly speaking, an efficient computation achieves the target accuracy in the shortest CPU time possible. The total compute time depends on the number of time steps used to cover the interval [T, T + ΔT^splitting], and on the CPU time spent in each of these steps.

We have seen that the solution of stiff chemistry requires implicit time integration algorithms. Most of the computational effort per step is spent in solving the system of nonlinear equations. In a Newton-Raphson approach, the LU factorization of the Jacobian is computed once, and is reused for all iterations; most of the computational effort is spent on performing the LU factorization and the repeated substitutions. The following ideas have proved successful in reducing the computational effort per step:

The reduction in the number of necessary time steps requires a good mechanism for time step adaptivity [25]. Such a mechanism should provide a sufficiently conservative error estimation to avoid a large number of step rejections, yet should be aggressive enough to quickly increase the time step after the transient and cover [T, T + ΔT^splitting] in a small number of steps.

A considerable increase in efficiency is possible based on the important observation that chemical systems Equation (2) need to be solved independently in each grid cell. Thus the overall chemistry computation is embarrassingly parallel [26], with as many independent tasks as there are grid cells in the spatial discretization. A direct parallel computation can be built via domain decomposition, where the computational grid is split into tiles, and each tile is mapped onto a different processor. Each processor solves chemistry in all gridpoints belonging to its associated tile [27–31]. An early approach to exploit parallelism was through vectorization [21], where each instruction line of the chemical solver acts repeatedly on different data items associated with different cells. This idea has found renewed interest due to modern accelerator architectures such as the Cell Broadband Engine and general purpose graphical processing units (GPGPUs) [32–35].

The QSSA method [51] is of historical significance since it was one of the earliest numerical schemes used to treat chemistry in air quality simulations. QSSA uses an approximate implicitness and avoids the solution of nonlinear systems completely.

Starting with the production-destruction form of the chemical kinetic ODE Equation (2), one keeps the arguments of P and D fixed at the current time step t_n. The resulting approximate ODE is linear y ′ = P ( t n , y n ) + D ( t n , y n ) y , t n ≤ t ≤ t n + 1 = t n + hwhere y_n is the numerical solution that approximates y(t_n) and h is the chemical integration step size (typically smaller than the operator splitting time step ΔT^splitting in Equation 2). This linear system can be solved analytically: (3) y n + 1 = e − h D ( t n , y n ) y n + ( I d − e − h D ( t n , y n ) ) D − 1 ( t n , y n ) P ( t n , y n )where I_d ∈ ℝ^d×d is the identity matrix. Owing to the diagonal structure of D(t_n+1, y_n+1), the QSSA formula Equation (3) is solved component by component, i.e., it involves only scalar operations. QSSA can be interpreted as the exponentially fitted Euler method [52], with the Jacobian J(t, y) = f_y(t, y) approximated by the diagonal destruction matrix D(t, y).

A careful analysis of the QSSA method has revealed that it is of first order, and improved QSSA methods remain of first order under stiffness [20]. QSSA provides positive solutions, but it does not preserve the linear invariants of the system (i.e., total mass and charge). Since the calculations are done component by component, the QSSA implicitness does not account for fast interactions happening among multiple species. QSSA is stable when the eigenvectors associated with the stiffest eigenvalues are close to unit vectors. When this is not the case, QSSA requires considerable reductions of the step size for stability. KPP offers implementations of several versions of the QSSA method.

Backward differentiation formulas (BDF) have become famous under the name “Gear” methods for solving chemical kinetic problems. BDF are linear multistep methods with excellent stability properties for the integration of stiff systems [15]. BDF methods have been applied extensively in chemical transport modeling. An important instance is the celebrated SMVGEAR (sparse matrix vectorized Gear [21]) code. Examples of representative air pollution models using SMVGEAR or SMVGEAR II code include CMAQ, GEOS-CHEM and GATOR-GCMO [40]. High quality, general purpose implementations of BDF methods are provided by the codes LSODE (Livermore Solver for ODEs, [53]), VODE (Variable coefficient ODE solver [54]), and Sundials (suite of nonlinear and differential/algebraic equation solvers [55]). The closely related algorithms NDF (numerical differentiation formulas) are implemented by the ode15 s stiff ODE solver in Matlab.

The k-step BDF method reads [15] (4) ∑ i = 0 k α i y n + 1 − i = h β f ( t n + 1 , y n + 1 )where the coefficients α_i and β are particular to the method and ensure that the order of consistency is k. The order k varies between one and five; higher order BDF formulas are unusable due to their lack of stability.

Practical implementations of BDF formulas are able to adapt both the time step and the order to achieve maximum efficiency For easily adjusting the step size it is convenient to represent the past history by the Nordsieck array [56], as is done in LSODE (5) z n = [ y n , h y ˙ n , … , h k y n ( k ) / k ! ] ∈ ℝ d × ( k + 1 )where, instead of storing y_n+1−k,…, y_n, one stores the derivatives of the solution at the current time. (The solution reconstruction is based on a Taylor polynomial approximation, rather than on a polynomial interpolant.) A step size change from h to rh is accommodated through a simple rescaling of the entries of z_n by powers of r.

The nonlinear system Equation (4)—in Nordsieck formulation—is solved for y_n+1 by a Newton-Raphson iterative approach. The starting point provided by the k^th order “predictor” approximation of z_n is given by (6) z n [ 0 ] = z n − 1 Awhere the transformation matrix A is a lower-triangular Pascal matrix [56]. The product z_n−1 A can be carried out by repeated additions to reduce the considerable computational effort. To save memory, A need not be stored and Darshana overwrites z_n−1 directly.

The Newton-Raphson iterations proceed as follows (7) g ( y n + 1 [ m ] ) = h f ( t n + 1 , y n + 1 [ m ] ) − h y ˙ n + 1 [ 0 ] − e n + 1 [ m ] e n + 1 [ m + 1 ] = e n + 1 [ m ] + P − 1 g ( y n + 1 [ m ] ) , m = 0 , … , M − 1 z n + 1 = z n + 1 | [ 0 ] + e n + 1 [ m ] [ 1 , ℓ 1 , … , ℓ k ]where ℓ_i are appropriate coefficients. Note that the first column of z_n+1 is the ODE solution y_n+1. The “prediction” matrix is P = I − h β J(t_n, y_n). In order to reduce the number of LU factorizations, the step size and the Jacobian can be held constant for multiple steps, in which case P = I − h_n−j β J(t_n−j, y_n−j) with j > 0.

After the iterations converge, the local truncation error is estimated by (8) d n + 1 = k ! ℓ k k + 1 e n + 1 [ M ]For step size control one defines a scaled norm of the local truncation error (9) Err = ‖ d n + 1 ‖ = 1 d ∑ i = 1 d ( d i , n + 1 RelTol i ⋅ max { | y i , n | , | y i , n | } + AbsTol i ) 2where RelTol and AbsTol are user-supplied vectors of relative and absolute error tolerances that describe the target accuracy level. For a 3D chemical transport model it is necessary to use vector, rather than scalar, absolute tolerances. Moreover, the concentrations of different species vary considerably across grid points; for example, the pressure change with altitude leads to orders of magnitude differences in the number of molecules per cubic centimeter. As a result, constant absolute error tolerances are not appropriate in three dimensional simulations [57]. Absolute tolerance vectors should account for different concentration levels of different species within the same grid point, and should vary with location to capture the geographic and altitude variation of concentrations.

If ‖d_n+1‖ ≤ φ_safe the solution y_n is accepted, otherwise it is rejected for being insufficiently accurate. The safety factor has a value slightly smaller than one, typically φ_safe = 0.9. In both situations a change in stepsize and/or order is considered in order to maximize computational efficiency.

Note that VODE uses a variable-coefficient implementation (fixed-leading coefficient form) instead of the fixed-step-interpolate methods in LSODE. The fixed-leading coefficient form shows better performance on many, though not all, stiff problems. KPP offers interfaces to both LSODE and VODE modified to use the optimized sparse linear algebra routines generated by KPP.

A general s-stage implicit Runge-Kutta method reads [15] (10) k i = y n + h ∑ j = 1 s a i j f ( t n + c j h , k j ) , i = 1 , 2 , … , s y n + 1 = y n + h ∑ j = 1 s b j f ( t n + c j h , k j ) where the coefficients a_ij, b_i and c_i define the method and are are chosen such that the desired accuracy and stability properties are obtained. To reduce the influence of round-off errors, implementations use the variable transformation z_i = k_i − y_n [15] in Equation (10) to obtain the equivalent form (11) z i = h ∑ j = 1 s a i j f ( t n + c j h , y n + z j ) , i = 1 , 2 , … , s y n + 1 = y n + ∑ i = 1 s d i z i The nonlinear system Equation (11) is solved at each step to obtain z₁,…, z_s. For general, fully implicit Runge-Kutta methods this system is of dimension ds × ds [41] (12) [ z 1 z 2 ⋮ z s ] = ( A ⊗ I d ) ⋅ [ f ( t n + c 1 h , y n + z 1 ) f ( t n + c 2 h , y n + z 2 ) ⋮ f ( t n + c s h , y n + z s ) ]where A = (a_ij) is the matrix of method coefficients, and ⊗ is the matrix Kronecker product. The Kronecker product of P ∈ ℝ^m×m and Q ∈ ℝ^n×n is the following mn × mn matrix (13) P ⊗ Q = [ p 11 Q p 12 Q ⋯ p 1 n Q p 21 Q p 22 Q ⋯ p 2 n Q ⋮ ⋮ ⋱ ⋮ p m 1 Q p m 2 Q ⋯ p n n Q ]Using the compact notation (14) Z = [ z 1 z 2 ⋮ z s ] , F ( Z ) = [ f ( t n + c 1 h , y n + z 1 ) f ( t n + c 2 h , y n + z 2 ) ⋮ f ( t n + c s h , y n + z n ) ]the nonlinear system Equation (12) can be written as (15) Z = ( A ⊗ I d ) ⋅ F ( Z )The system Equation (15) is solved by simplified Newton-Raphson iterations [15] of the form (16) [ I d s − h A ⊗ J ( t n , y n ) ] Δ Z [ m ] = Z [ m ] − ( h A ⊗ I d ) F ( Z [ m ] ) Z [ m + 1 ] = Z [ m ] − Δ Z [ m ] , m = 0 , … , M − 1Note that the Jacobian is only evaluated at the beginning of the current time step. Following [15], a transformation of the system Equation (16) to complex form replaces the costly ds-dimensional real LU decomposition by several d-dimensional LU decompositions of real and complex matrices.

A Singly Diagonally-Implicit Runge-Kutta (SDIRK) method is a special case of the fully implicit Runge-Kutta method with coefficients satisfying a_ij = 0 for j > i and a_ii = γ for all i. In contrast to the fully implicit Runge-Kutta method, the nonlinear system Equation (12) naturally decouples into a sequence of d-dimensional real systems [41] of the form (17) z i = h ∑ j = 1 i − 1 a i j f ( t n + c j h , y n + z j ) + h γ f ( t n + c i h , y n + z i )Each stage i solves for the vector z_i by simplified Newton-Raphson iterations (18) [ I d − h γ J ( t n , y n ) ] Δ z i [ m ] = z i [ m ] − h ∑ j = 1 i − 1 a i j f ( t n + c j h , y n + z j ) z [ m + 1 ] = z i [ m ] − Δ z i [ m ] , m = 0 , … , M − 1The Jacobian matrix is only evaluated at the beginning of the time step. The LU factorization of I_d − h γ J(t_n, y_n) is shared for all iterations m and for all stages i, so that only one LU decomposition is performed in each time step.

An estimator of the local truncation error is obtained with the help of the embedded formula (19) y ^ n + 1 = y n + ∑ i = 1 s b ^ i k iThis formula provides an alternative numerical solution ŷ_n+1 using the already computed increment vectors k_i, but different weights b̂_i. The coefficients are usually chosen such that the order of consistency of ŷ_n+1 is one less than that of y_n+1. The difference vector d_n+1 = ŷ_n+1 − y_n+1 provides the local error estimator. As pointed out in [41], such estimators work well with SDIRK methods, but they are more difficult to construct for fully implicit Runge-Kutta methods since additional stages are needed.

The step adjustment strategy uses Equation (9) to compute the error norm Err = ‖d_n+1‖ based on the user specified relative and absolute tolerances. The step is accepted if Err ≤ φ_safe, and rejected otherwise. A rejected step is repeated with a smaller step size. The safety factor has a value slightly smaller than one, typically φ_safe = 0.9.

The new step size is estimated by the asymptotic formula (20) η = min ( ϕ max , max ( ϕ min , ϕ safe ⋅ E r r − 1 / ( p ^ + 1 ) ) ) , h new = h old ⋅ ηwhere φ_max is an upper bound, and φ_min a lower bound on the step change factor. Typical values are φ_max = 10, φ_min = 0.1, and φ_safe = 0.9. If the step size has been recently rejected, the allowed increase factor is further limited (e.g., φ_max = 1 following a rejection). Furthermore, the step size is constrained such that h_min ≤ h ≤ h_max, and the starting step size is specified h = h_start. Numerical experiments in [41–43] demonstrate that this step size control strategy works well for a wide range of atmospheric chemical kinetic problems.

Several Runge Kutta methods are available in the KPP numerical library. The fully implicit schemes implemented are the 3-stage Radau-IIa, Radau-Ia, Lobatto-IIIc, and Gauss methods [15]. The SDIRK schemes involve Sdirk-4a and Sdirk-4b (5 stages, order 4, L-stable), Sdirk3a (3 stages, order 2, stiffly accurate), and Sdirk2a and Sdirk-2b (2 stages, order 2, stiffly accurate).

Rosenbrock methods are competitive with other stiff solvers for low to modest accuracy, and therefore are attractive for atmospheric chemistry applications [23]. Rosenbrock methods can be considered as linearly-implicit versions of Runge Kutta methods. To avoid nonlinear systems, the Jacobian is used directly in the integration formula [15,58] and each stage requires the solution of a linear system. For example, the backward (fully implicit) Euler method solves at each step the nonlinear system y n + 1 = y n + h f ( y n + 1 )to obtain y_n+1. The linearly implicit Euler method solves a linear system to obtain an increment vector k, which is then used to update the solution (21) k = h f ( t n , y n ) + h J ( t n , y n ) ⋅ k , y n + 1 = y n + kThis algorithm corresponds to a single Newton iteration solving the backward Euler equation. Note that the system Jacobian matrix J appears explicitly in the discretization scheme.

A general s-stage Rosenbrock method Rosenbrock [15,58] computes the next step solution as follows: (22) k i = h f ( t n + α i h , y n + ∑ j = 1 i − 1 α i j k j ) + γ i h 2 f t ( t n , y n ) + h J ( t n , y n ) ⋅ ∑ j = 1 i γ i j k j y n + 1 = y n + ∑ i = 1 s b i k iwhere the coefficients b_i, α_ij and γ_ij define particular methods and are chosen such as to obtain the desired accuracy and stability properties [22]. We have that α_ij = 0 for j ≥ i, γ_ij = 0 for j ≥ i + 1, and α i = ∑ j = 1 i − 1 α i j , γ i = ∑ j = 1 i γ i jThe term f_t represents the partial derivative of the ODE function with respect to time, and is equal to zero in the case of autonomous systems.

For implementation purposes, it is advantageous to choose all the diagonal coefficients equal to each other, γ_ii = γ for all stages i = 1,…, s, and to avoid Jacobian-vector products by changing Equation (22) to the mathematically equivalent formulation [15] (23) ( 1 h γ I − J ( t n , y n ) ) ⋅ u i = f ( t n + α i h , y n + ∑ j = 1 i − 1 a i j u j ) + ∑ j = 1 i − 1 c i j h u j + h γ i f t ( t n , y n ) y n + 1 = y n + ∑ j = 1 s m j u jwhere Γ = ( γ i j ) 1 ≤ i , j ≤ s , ( a i j ) 1 ≤ i , j ≤ s = ( α i j ) 1 ≤ i , j ≤ s ⋅ Γ − 1 , ( m i ) 1 ≤ i ≤ s = ( b i ) 1 ≤ i ≤ s ⋅ Γ − 1Note that Equation (23) avoids not only the matrix-vector multiplications, but also the n² multiplications for constructing the matrix-scalar product hγJ. The computational cost per step for the Rosenbrock method Equation (23) is that of one real d × d LU factorization, s forward and backward substitutions, and one Jacobian plus s function evaluations. This cost is similar to that of one BDF step using M = s simplified Newton iterations Equation (7) and no Jacobian factorization re-use.

The local error estimator for Rosenbrock methods is based on an embedded formula for error estimation, similar to Equation (19). The step is accepted if the error norm Equation (9) is below φ_safe = 0.9, and rejected otherwise. The next time step size is calculated by Equation (20).

An important sub-class are the Rosenbrock-W methods, which allow the use of any approximation J̃ instead of the complete Jacobian J in Equation (23) without losing their order of accuracy. Rosenbrock-W methods may result in considerable computational savings by replacing the Jacobian with a sparser approximation, with a tensor product of smaller matrices, etc.

Careful benchmarks of stiff solvers [23,24,42,59] indicate that Rosenbrock methods are the most efficient for the low to medium accuracy range required in chemical transport applications.

Several Rosenbrock methods are available in the KPP numerical library. They are Rodas (the 6-stage method based on a stiffly accurate pair of order 4(3) [15]), Ros4 (based on a 4 stage, L-stable, embedded pair of order 4(3) [15]), Rodas3 (a 4-stage, stiffly accurate, embedded pair of order 3(2) [23]), Ros3 (a 3-stage, L-stable pair of order 3(2) [23]), and Ros2a and Ros2b (2-stage stiffly accurate pairs of order 2(1) [23]). In addition, the method Rang3 is a Rosenbrock-W scheme of order 3 with 5 stages.

This family of methods constructs numerical solutions by applying Richardson extrapolation to a sequence of low order approximations, each made with a different step size [15]. The extrapolation approach can be used to construct methods of arbitrarily high order. Extrapolation methods are very effective for high accuracy calculations.

Consider a sequence of step sizes τ₁, τ₂, τ₃, … defined by by τ_j = h/j. Further, consider the linearly implicit Euler method Equation (21) as the “base” numerical method for solving the system Equation (2). Denote by (24) T j , 1 : = y τ j ( t n + h ) , 1 ≤ j ≤ pthe numerical approximation of y(t_n+1) obtained as follows. Start from y_n, and march over the interval [t_n, t_n+1] using j steps of Equation (21), each with a step size τ_j. All steps use the Jacobian J(t_n, y_n), and therefore share the same LU decomposition. The global error of the linearly implicit Euler method has an asymptotic expansion of the form (25) y ( t n + 1 ) − y h ( t n + 1 ) = e 1 ( t n + 1 ) τ 1 + ⋯ + e N ( t n + 1 ) τ N + E τ ( t n + 1 ) τ N + 1where e_i(t_n+1) are errors that do not depend on τ, and E_τ is bounded for t_n ≤ t ≤ t_n+1. (For very stiff problems a different, perturbed expansion of the global error holds [15], but this does not change the fundamental idea of this discussion). Using the M approximations Equation (24) obtained with different h_j's, the error terms in the global error asymptotic expansion Equation (25) can be eliminated by recursive Richardson extrapolation [60] to build new approximations: (26) T j , k + 1 = j T j , k − ( j − 1 ) T j − 1 , k , j ≤ p , k < jThe numerical scheme Equations (24)–(26) is called the extrapolation method, and it builds an entire family of approximations. Each scheme T_j,k+1 is of order k, since extrapolation eliminates the terms e1,…, e_k from the global error expansion Equation (25). We remark that extrapolation provides a sequence of lower-order embedded method, which can be used for step size and order control.

The KPP numerical library offers an interface to the SEULEX code [15] modified to use the optimized sparse linear algebra routines generated by KPP.

In a chemical kinetic solver, most of the computational effort is spent in solving the linear systems associated with the implicit time integration algorithms. For all methods discussed here the matrix of coefficients is of the form I − h γ J, and inherits the sparsity structure of the Jacobian.

In a typical chemical mechanism, the pattern of chemical interactions leads to a Jacobian that has the majority of entries equal to zero. Figure 1 displays the sparsity structure of the Jacobian of the MECCA chemical mechanism, used in the numerical experiments presented in Section 5. This Jacobian has 2655 nonzero entries, i.e., 5.6% of its total number of elements.

Linear algebra algorithms can take advantage of this to avoid unnecessary operations and greatly reduce CPU time. Since the sparsity structure depends only on the chemical network (and not on the values of concentrations or rate coefficients) it can be computed offline [22,61]. This approach has been taken by SMVGEAR [21] and by the Kinetic PreProcessor KPP [25]. KPP prepares highly efficient routines for sparse LU decomposition and substitution that are specific to the particular sparsity of the simulated chemical mechanism [24]. All stiff numerical methods implemented in the KPP library make use of these routines.

Recent developments inmulti-core chipset architectures canbe leveraged to reduce chemical simulation runtime. In general, good performance is achieved by using every tier of heterogeneous parallelism available to the model. Chemical kinetics are embarrassingly parallel between grid cells, so there is abundant data parallelism (DLP). Within the solver itself, the ODE system is coupled so that, while there is still some data parallelism available in lower-level linear algebra operations, parallelization is limited largely to the instruction level (ILP). (Some specific chemical mechanisms are only partially coupled and can be separated into a small number of sub-components, but such inter-module decomposition is rare.) Thus, a three-tier parallelization is possible: ILP on each core, DLP using single-instruction-multiple-data (SIMD) features of a single core, and DLP across multiple cores (using multi-threading) or nodes (using MPI). The coarsest tier of MPI and OpenMP parallelism is typically supplied by the atmospheric model.

This section presents parallelization strategies for Rosenbrock integration in one-cell-per-thread, N-cells-per-thread, and 4/2-cells-per-thread decompositions. For performance benchmarks of parallelized Rosenbrock solvers, see [32–34,62].

KPP makes it possible to rapidly generate correct and efficient chemical kinetics solvers on scalar architectures, but these generated codes cannot be easily ported to multi-core accelerated or heterogeneous architectures. KPPA (the Kinetics PreProcessor: Accelerated) [32], is the next generation KPP tool that achieves significantly reduced time-to-solution for chemical kinetics kernels on both traditional and emerging architectures. In addition to the basic KPP functionality, KPPA generates OpenMP code with SSE or Alitivec for traditional CPUs, CUDA code for NVIDIA GPUs, and optimized C codes for the Cell Broadband Engine Architecture (CBEA), in either double or single precision. KPPA-generated mechanisms leverage platform-specific multi-layered heterogeneous parallelism to achieve strong scalability. Compared to state-of-the-art serial implementations, speedups of 20×–40× are regularly observed in KPPA-generated code.

KPPA combines a general analysis tool for chemical kinetics with a code generation system for scalar, homogeneous multi-core, and heterogeneous multi-core architectures. It is written in object-oriented C++ with a clearly-defined upgrade path to support future multi-core architectures as they emerge. KPPA has all the functionality of KPP 2.1 and maintains backwards compatibility with KPP. Many atmospheric models, including WRF-Chem and STEM, support a number of chemical kinetics solvers that are automatically generated at compile time by KPP. Reusing these analysis techniques in KPPA insures its accuracy and applicability.

KPPA's code generation component accommodates a two-dimensional design space of programming language/target architecture combinations superseding the one-dimensional design space of KPP (Table 2). Given the model description from the analytical component and a description of the target architecture, the code generation component produces a time-stepping integrator, the ODE function and ODE Jacobian of the system, and other quantities required to interface with an atmospheric model.

KPPA's key feature is its ability to generate fully-unrolled, platform-specific sparse matrix/matrix and matrix/vector operations that achieve very high levels of efficiency. As KPPA parses it's input, language independent expression trees describing sparse matrix/matrix or matrix/vector operations are constructed in memory. For example, the aggregate ODE function of the chemical mechanism is calculated by multiplying the left-side stoichiometric matrix by the concentration vector, and then adding the result to elements of the stoichiometric matrix. KPPA performs these operations symbolically at code generation time, using the matrix formed by the analytical component and a symbolic vector, which will be calculated at run-time. The result is an expression tree of language-independent arithmetic operations and assignments, equivalent to a rolled-loop sparse matrix/vector operation, but in completely unrolled form.

KPPA uses its knowledge of the target architecture to generate highly-efficient code from the expression tree. Vector types are preferred when available, branches are avoided on all architectures, and parts of the function can be rolled into a tight loop if KPPA determines that on-chip memory is a premium. An analysis of four KPPA-generated ODE functions and ODE Jacobians targeting the CBEA showed that, on average, both SPU pipelines remain full for over 80% of the function implementation. Pipeline stalls account for less than 1% of the cycles required to calculate the function. For example, in the SAPRCNOV mechanism on CBEA, there are only 20 stalls in the 2989 cycles required by the ODE function (0.66%), and only 24 stalls in the 5490 cycles required for the ODE Jacobian (0.43%). Code of this caliber typically requires meticulous hand-optimization, but KPPA is able to generate this code automatically in seconds. See [32] for further performance analysis of KPPA.