Most regulatory applications of PGMs only evaluate model performance in an operational sense, i.e., they compare model estimates of ozone concentrations for the base year with measurements. They normally do not include a full dynamic evaluation, i.e., an evaluation of the ability of the modeling system (the model inputs and the model itself) to respond correctly to historical and recent changes in precursor emissions, due to time constraints, or due to difficulties in performing such an evaluation because of complications introduced by uncertainties in emission changes and meteorological variability [2
]. As noted by Hogrefe et al. [4
], an operational model performance evaluation does not necessarily provide information on how well the modeling system will perform in the regulatory setting of determining responses to emission changes. Hogrefe et al. [4
] recommend placing more emphasis on diagnostic and retrospective dynamic evaluation approaches than on operational performance alone.
The South Coast Air Basin (SoCAB) of California experiences some of the highest ozone concentrations in the U.S. with many exceedances of the National Ambient Air Quality Standards (NAAQS) for ozone [5
]. The high concentrations are a result of large precursor emissions from the greater Los Angeles urban area, trapping of pollutants by the marine inversion and mountains on three sides, and high temperatures and abundant sunlight to promote ozone formation photochemically from precursor emissions [6
]. The South Coast Air Quality Management District (SCAQMD) has the responsibility for implementing control measures to bring the region into compliance with the NAAQS. Stringent controls on VOC and NOx
emissions over the last 5 decades have resulted in a dramatic improvement in ozone levels in the SoCAB [7
]. Although the 2008 NAAQS is still exceeded in the Basin on many summer days, the year-to-year ozone reductions have been significant. However, attainment of the more stringent 2015 NAAQS in the future (beyond 2032) presents yet another challenge [8
As part of its efforts to bring the SoCAB region into compliance, the SCAQMD prepares an Air Quality Management Plan (AQMP) approximately every 4 years. The AQMP relies on photochemical grid modeling to determine the effectiveness of emission control measures. Inaccuracies in predicted model responses to emission controls are likely to lead to controls that are either ineffective (if the model over-estimates the response to emission changes) or too stringent (if the model under-estimates the response). Although an operational model performance evaluation for the base modeling year is conducted as part of the AQMP development, there is no assessment of the responsiveness of the model to emission changes over a long time period (i.e., several years). This paper describes a dynamic evaluation of the AQMP modeling system to determine how well the system responds to emissions changes in the SoCAB over a period of 20 to 25 years. A number of dynamic evaluation studies in other contexts have been conducted and those are discussed briefly in the following section.
Previous Dynamic Evaluation Studies
A recommended model evaluation framework [10
] consists of four components:
Operational evaluation: generate statistics of the deviations between model estimates for a simulation year and corresponding observations, and compare the magnitudes of those deviations to selected criteria
Diagnostic evaluation: test the ability of the model to simulate each of the interacting processes that govern the system
Dynamic evaluation: test the model’s ability to predict changes in air quality concentrations in response to changes in either source emissions or meteorological conditions
Probabilistic evaluation: focus on the modeled distributions of selected variables rather than individual model estimates at specific times and locations
Dynamic evaluation looks at retrospective cases to evaluate whether the model has properly predicted air quality responses to known emission and/or meteorological changes [10
]. The change in concentration is evaluated instead of the “base” concentration itself, unlike operational and diagnostic aspects of model evaluation. The ability of the model to reproduce historical pollution trends provides confidence in its use for making future year projections.
The U.S. EPA ozone and PM2.5
modeling guidance [11
] also discusses this model evaluation framework. For various reasons, including time and resource constraints, most performance assessments of the modeling that underlies regulatory decision-making focus primarily on the first evaluation component, i.e., the operational evaluation that tests how well the model reproduces concurrent observed air quality concentrations for the base year. However, that testing does not evaluate the model in the way it is used in regulatory planning to predict changes in future air quality based on estimates of changes in future emissions [13
In addition to resource constraints, challenges to conducting a true dynamic evaluation include the influence of year-to-year meteorological variability on the observed trends [10
] and the difficulties in developing modeling emission inventories for historical years [14
]. Alternative approaches to a multi-year dynamic evaluation include conducting an assessment of model performance for weekday/weekend concentration differences, where mobile source emissions are known to change significantly [10
], or assessing the model’s ability to capture the main time variations [14
] within the simulated period (e.g., weekly, day-night and/or seasonal). While those approaches provide useful information, they are less appropriate for evaluating a model’s accuracy for multi-year air pollution control planning.
Recognizing the importance of dynamic evaluation in making projections of future-year ozone, a number of dynamic evaluation studies with the U.S. EPA Community Multiscale Air Quality (CMAQ) modeling system [15
] have been conducted over the last decade, particularly in the eastern U.S. Those studies are discussed briefly here.
Gilliland et al. [16
] took advantage of the large NOx
emission reductions between 2002 and 2005 associated with the EPA’s NOx
State Implementation Plan (SIP) Call, in addition to a more gradual decreasing trend in mobile on-road emissions during that period, to assess the ability of CMAQ to predict changes in ozone. Large decreases in the measured daily maximum 8-h (MDA8) ozone concentrations were reported between 2002 (pre-NOx
SIP Call period) and 2004 and 2005 (post-NOx
SIP Call period). The observed decreases from 2002 to 2004 in O3
levels were larger than the decreases between 2002 and 2005, because the summer of 2004 was less conducive to ozone formation than the summers of 2002 and 2005 due to cooler and wetter conditions during 2004 [16
]. The similar meteorological conditions between 2002 and 2005 better isolated the influence of emission reductions on O3
concentrations, because NOx
emissions in 2004 and 2005 were comparable and considerably lower than 2002 emissions. CMAQ (versions 4.5 and 4.6) simulations conducted for the summer periods (June through August) of 2002 and 2005 showed that the model-predicted decrease in MDA8 O3
was less than the observed decrease using 3 different atmospheric chemistry mechanisms (Carbon Bond 4 (CB4); Carbon Bond 2005 (CB05); Statewide Air Pollution Research Center 1999 (SAPRC-99)), although the CB05 mechanism was incrementally better than the other two mechanisms [16
]. The comparisons between 2002 and 2004 also showed under-predictions of the ozone reductions from 2002 to 2004, but a significant part of the ozone decrease was due to the differences in meteorology between the two years and the SAPRC mechanism captured the O3
differences better than the CB4 mechanism. Gilliland et al. [16
] suggest that a number of factors may have contributed to the slower response of the model to emission changes, including errors in the NOx
emission inputs or under-prediction of long-range transport of ozone and its precursors.
Pierce et al. [17
] conducted a dynamic evaluation of CMAQ using weekend-weekday (WEWD) differences in ozone precursor emissions and 18 years of modeled and observed ozone concentrations. They found that the modeled response of ozone to WEWD differences in emissions was less than the observed response. They attributed the lower response to uncertainties in mobile source NOx
emissions and boundary conditions, as well as to grid resolution.
Napelenok et al. [2
] extended the analysis of Gilliland et al. [16
] by conducting a dynamic evaluation of CMAQ v4.7.1-predicted ozone changes due to the NOx
SIP Call in the eastern U.S. between 2002 and 2005, and explicitly accounting for known uncertainties in the NOx
emissions inventories. They considered uncertainty in three NOx
emission sectors: area sources, mobile sources, and point sources. Assuming moderate (50%) uncertainty in area and mobile source NOx
emissions and a small uncertainty (3%) in the utility sector, they found that the model was able to reproduce the observed changes in MDA8 O3
concentrations at more than two-thirds of the monitoring locations. Assuming larger uncertainties (100%) in area and mobile source NOx
emissions, the observed change in the ozone distribution was captured at more than 90% of the sites. Other sources of uncertainty (boundary conditions, VOC emissions, chemistry) were also found to have an impact on model response.
Similarly, Zhou et al. [18
] conducted an evaluation of CMAQ 4.7-predicted ozone changes for the NOx
SIP Call region between 2002 and 2006. As in the previous studies [16
], it was found that observed downward changes in mean NOx
(−11.6 to −2.5 ppb) and 8-h O3
(−10.4 to −4.7 ppb) concentrations in metropolitan areas in the NOx
SIP Call region were under-predicted by the CMAQ model by 31% to 64% and 26% to 66%, respectively. Sensitivity studies showed that the under-prediction in O3
improvements could be alleviated by 5% to 31% by constraining NOx
emissions in each year based on observed NOx
concentrations, while adjusting for temperature biases in the meteorological input [18
]. Focusing on uncertainties in the chemical reaction rate constants had a smaller influence on the predicted responses.
All of the foregoing dynamic evaluation studies, based on CMAQ versions 4.x, concluded that the modeling system underestimated the observed ozone reductions after the implementation of the NOx
SIP Call, and that modeled ozone responses could be improved by adjusting ground-level NOx
emission inputs, but, even then, observed ozone reductions were still under-estimated. More recently, Foley et al. [1
] assessed the impacts of the model updates included in CMAQ v5.01 on the dynamic evaluation of ozone predictions for the 2002–2005 NOx
SIP Call period. While the median bias for high summertime ozone decreased in both years compared to previous simulations with CMAQ v4.x, the observed decrease in ozone from 2002 to 2005 in the eastern US continued to be underestimated by the model [1
]. Sensitivity studies showed that emission controls led to a decrease in modeled high summertime ozone that was nearly twice as large as the decrease attributable to changes in meteorology alone, indicating that the model response to emission reductions during the NOx
SIP Call period continued to be lower than the observed response, even with the updates to the model [1
The dynamic evaluation studies cited above involved regional-scale modeling with grid resolutions of the order of a few kilometers. CMAQ was also used in a hemispheric simulation [20
] with a grid resolution of 108 km to model air quality trends across the Northern Hemisphere over a period of 20 years (1990–2010). The air quality simulations were driven by year-specific meteorological fields simulated by the Weather Research and Forecasting (WRF) model and year-specific global anthropogenic emission inventories obtained from the Emission Database for Global Atmospheric Research (EDGAR) for 1990–2008. For 2009 and 2010, emissions for the U.S., Europe and China for 2009 and 2010 were based on the most recent available projections for those years, while emissions for other areas were kept at 2008 levels. The modeled decrease in the annual maxima of daily maximum 8-h average ozone in the eastern U.S. was about 0.5 ppb/year, while the observed decrease was nearly a factor of two higher at about 0.9 ppb/year [20
The most recent public release of CMAQ (v5.1) incorporates a large number of scientific updates and extended capabilities over the previous release version of the model (v5.0.2). While dynamic evaluation studies with this updated version of CMAQ are not yet published, Appel et al. [21
] conducted sensitivity studies for several hypothetical emission reduction scenarios, and found that CMAQ v5.1 tends to be more responsive to reductions in NOx
emissions in predicting ozone reductions than v5.0.2. Appel et al. [21
] suggest that this represents an improvement over previous versions of CMAQ, which underestimated O3
reductions in response to large, widespread emission reductions, as discussed in the studies cited above.