Next Article in Journal
Maternal Undernutrition and Breast Milk Macronutrient Content Are Not Associated with Weight in Breastfed Infants at 1 and 3 Months after Delivery
Next Article in Special Issue
Development of TracMyAir Smartphone Application for Modeling Exposures to Ambient PM2.5 and Ozone
Previous Article in Journal
The Effects of Tocotrienol on Bone Peptides in a Rat Model of Osteoporosis Induced by Metabolic Syndrome: The Possible Communication between Bone Cells
Previous Article in Special Issue
African American Exposure to Prescribed Fire Smoke in Georgia, USA
 
 
Due to planned maintenance work on our platforms, there might be short service disruptions on Saturday, December 3rd, between 15:00 and 16:00 (CET).
Order Article Reprints
Font Type:
Arial Georgia Verdana
Font Size:
Aa Aa Aa
Line Spacing:
Column Width:
Background:
Article

Application of a Fusion Method for Gas and Particle Air Pollutants between Observational Data and Chemical Transport Model Simulations Over the Contiguous United States for 2005–2014

1
School of Civil and Environmental Engineering, Georgia Institute of Technology, Atlanta, GA 30332, USA
2
Department of Biostatistics and Bioinformatics, Rollins School of Public Health, Emory University, Atlanta, GA 30322, USA
*
Author to whom correspondence should be addressed.
Int. J. Environ. Res. Public Health 2019, 16(18), 3314; https://doi.org/10.3390/ijerph16183314
Received: 14 July 2019 / Revised: 2 September 2019 / Accepted: 4 September 2019 / Published: 9 September 2019
(This article belongs to the Special Issue Air Quality and Health Predictions)

Abstract

:
Accurate spatiotemporal air quality data are critical for use in assessment of regulatory effectiveness and for exposure assessment in health studies. A number of data fusion methods have been developed to combine observational data and chemical transport model (CTM) results. Our approach focuses on preserving the temporal variation provided by observational data while deriving the spatial variation from the community multiscale air quality (CMAQ) simulations, a type of CTM. Here we show the results of fusing regulatory monitoring observational data with 12 km resolution CTM simulation results for 12 pollutants (CO, NOx, NO2, SO2, O3, PM2.5, PM10, NO3, NH4+, EC, OC, SO42−) over the contiguous United States on a daily basis for a period of ten years (2005–2014). An annual mean regression between the CTM simulations and observational data is used to estimate the average spatial fields, and spatial interpolation of observations normalized by predicted annual average is used to provide the daily variation. Results match the temporal variation well (R2 values ranging from 0.84–0.98 across pollutants) and the spatial variation less well (R2 values 0.42–0.94). Ten-fold cross validation shows normalized root mean square error values of 60% or less and spatiotemporal R2 values of 0.4 or more for all pollutants except SO2.

1. Introduction

Fine scale air pollution fields with complete spatiotemporal coverage are of importance to public health researchers who aim to understand the drivers of adverse health effects [1,2]. Health exposure studies involve matching ground monitor air pollution measurements to individuals within the nearby vicinity of an air pollution monitor [3,4,5]. These measured concentrations are then used to develop a city-wide health effect estimate [6]. However, this method is restricted by the limited spatial monitoring networks across the contiguous United States, the impacts of local emissions on a single monitor, and the temporal completeness of the air monitor data [7,8,9].
One method to address the sparseness and temporal incompleteness of air monitoring data uses chemical transport model (CTM) results. CTMs use the governing chemical and physical principals with meteorology and emissions data to simulate pollutant concentrations [10,11,12]. One widely used CTM is the community multiscale air quality (CMAQ) model. Although CTMs can create spatially and temporally complete pollutant fields, there are biases in the model results [13,14,15]. Such biases differ based on the species simulated and the domain [10]. Examples of this bias include over-predictions of particulate nitrate and ammonium and underpredictions of carbonaceous aerosols in the spring and summer [11]. For the ozone these biases include an underestimation in North America of daytime ozone in the spring and an overestimation of daytime concentrations during the summer [10,16]. Evaluation of CMAQ to the ozone monitoring instrument (OMI) found that CMAQ overestimates summertime nitrogen dioxide (NO2) in urban locations and underestimates in rural locations [17]. As CMAQ and CTMs improve and the biases are better understood, there is less of a need to modify CTM results for use in exposure studies [18,19].
CTM simulations can be combined with air quality observations through data fusion (DF) methods to other air quality data to correct for the biases associated with the model outputs [20,21]. Models can be fused through mathematical methods such as statistical interpolation or machine learning approaches [22,23,24,25,26,27,28]. The data fused with CTM outputs can range from satellite, observational, land use, or meteorology data. Interpolation methods such as ordinary kriging and inverse distance weighting (IDW) can interpolate a parameter over space, while machine learning methods such as neural network, random forest, and decision trees create a model using a set of preselected variables [29]. The method selected depends on the pollutant being studied and the type of data available [30,31]. For example, speciated particulate matter may only have observational data available, while particulate matter less than equal to 2.5 microns (PM2.5) has satellite data derived aerosol optical depth (AOD) and observational data available [32,33,34]. Each method is associated varying performance depending on the type of data and method being used [35].
The approach of this study is to use a statistical DF method to combine observational data to the CMAQ model simulations to create accurate spatiotemporally resolved air pollution fields. We modified an existing data fusion method (Friberg et al) and applied it regionally across the contiguous United States at a 12 km resolution for 12 pollutants: seven particulate matter species and five gaseous species over a span of ten years [20]. The pollutants chosen were those linked to adverse health effects and that which are currently measured by the United States Environmental Protection Agency (USEPA) [20,36]. These pollutants are PM2.5, particulate matter with an aerodynamic diameter less than or equal to 10 microns (PM10), nitrate (NO3), ammonium (NH4+), sulfate (SO42−), elemental carbon (EC), organic carbon (OC), carbon monoxide (CO), nitrogen dioxide (NO2), nitrogen oxides (NOx), sulfur dioxide (SO2), and ozone (O3). Cross validation by data withholding is then used to evaluate the DF method. This evaluation requires comparing the concentrations produced through the DF method to the available measurement values.

2. Materials and Methods

2.1. CMAQ Source

The CMAQ runs were performed by the USEPA using version 5.0.2 of CMAQ at a 12 km resolution across the contiguous United States for the years 2005–2014, for 12 pollutants of interest, to produce an hourly simulation [37,38]. The pollutants modeled were 7 particulate species (PM10, PM2.5, EC, OC, NH4+, SO42−, NO3) and 5 gases (CO, NO, NOx, O3, SO2). CMAQ relies on process-based principals to predict concentrations of species while taking into account emissions, meteorology, and chemical/physical properties [10,11]. CMAQ was run with the bidirectional NH3 air–surface exchange and the Carbon Bond version 2005 (CB05) chemical mechanism [37,38]. The simulations were run at a 12 km × 12 km resolution with 35 vertical layers that span till the top of the free troposphere, with layer 1 nominally 19 m tall [37]. Emission inputs were based on 2005, 2008, and 2011 National Emission Inventory (NEI) using the Sparse Matrix Operator Kernel version 3.1 [37,39]. A year-specific continuous emission monitoring system for large combustion and industrial process was also included [37,39]. A more detailed description of the input information is available through the USEPA (Section S1). The surface level simulations were used in this DF method.
At a 12 km resolution, the contiguous United States is represented by a 299 × 459 cell matrix, with each cell centroid described through a latitude and longitude coordinate. The CMAQ simulation outputs hourly concentrations which are then converted to 24 h averages for particulate matter species, 1-h max average for gaseous species, and 8-h max average for ozone. These averaging methods were chosen based on the averaging methods of the current Environmental Protection Agency (EPA) standards of these pollutants [40,41]. Hourly concentrations from CMAQ at Greenwich Mean Time (GMT) were converted to the respective United States local time zone. The averaging was then performed from midnight to midnight local time. The time zone of each grid cell was determined through the latitude and longitude coordinates of the grid centroid.

2.2. Monitoring Data

The observational data was taken from the EPA’s Air Quality System (AQS) database for the same 2005–2014 time span and 12 pollutants [42]. This database contains observational air quality monitoring data from state, local, and tribal air pollution monitors across the contiguous United States [42]. Species such as ozone and PM2.5 had the best spatial coverage, while speciated PM had limited spatial coverage that was limited to metropolitan areas. Figure 1 shows a comparison between the spatial coverage of PM2.5 and EC across the contiguous United States.
The particulate matter species monitors sampled at three different frequencies: 1 in 6 days, 1 in 3 days, and daily. Monitors with daily measurements are only available for PM2.5 and PM10. The speciated PM (EC, OC, SO42−, NO3, NH4+) have monitors sampling 1 in 6 and 1 in 3 days. Table 1 shows the summary statistics for each of the species. The AQS PM data were downloaded as 24-hr averaged concentrations, the gaseous species (excluding ozone) were downloaded as 1-h max concentrations, and ozone was downloaded as 8-h max concentrations [42].

2.3. Data Fusion

To compensate for the monitoring data that are spatially and temporally incomplete, we fused the monitoring data to the CMAQ simulations. These simulations are independent of the monitoring data, based on emission estimates, meteorological data, and physical/chemical process modeling [10,11]. The first step is estimating two regression parameters α and β to correct the CMAQ simulation (Equation (1)).
C M A Q c o r r e c t e d = α C M A Q β
To estimate these parameters, we fit the above power regression model using annual averaged CMAQ ( C M A Q ¯ ) as predictors of annual averaged observational ( O B S ¯ ) data. The annual averaged CMAQ values were taken at the cell locations corresponding to the observational data and only when observational data points were available at each location (Equations (2) and (3)). This is of importance for non-daily or seasonal monitors that only take measurements for certain portions of the year.
O B S ¯ = 1 n i   m O B S i
C M A Q ¯ = 1 n i   m C M A Q i
Here, n denotes the total number of observations taken at a point, m represents the days in the year in which the monitor takes a measurement, the overbar represents the annual average, and the subscript i represents the particular day. The β coefficient is limited between 0 and 1 to prevent any exponential increasing functions. The α and β coefficients are calculated for each year of each species to account for yearly changes between the CMAQ and observational data. The regression parameters are then used to adjust all CMAQ values within the matrix. Some scatter is expected in this model because of the spatial misalignment of the point source monitor and the gridded CMAQ field. The next step is to convert the measurements to a normalized dimensionless ratio (Equation (4)); this is the term that is spatially interpolated.
O B S O B S ¯ · α C M A Q ¯ β α C M A Q β
The α and β terms represent the regression parameters found through the power regression (Equation (1)), O B S is the daily observational value, O B S ¯ is the annual averaged observational data, C M A Q is the value at a cell with an observation, and C M A Q ¯ is the annual averaged simulated value at that cell. Rather than only normalizing by the average of the observations, the inclusion of the normalized CMAQ term accounts for monitors with only a partial year set of measurements. The CMAQ values are averaged only when observations are available. If multiple observations are included in one CMAQ grid cell, the average of the observations is used. For species such as elemental and organic carbon which only record observational values every 1 in 3 or 1 in 6 days, we used linear interpolation temporally to calculate the normalized dimensionless ratio on days with no observational data. This allows us to use the existing observational data to estimate the concentrations on days without any data available. Table 2 shows the range of α and β values calculated for each species over the ten-year period.
The α and β were calculated for each year to allow for variations in the relationship between the yearly CMAQ results and observational data, rather than finding one set of parameters to describe all ten years.
This normalized ratio is calculated at each monitoring location. These point values are then spatially interpolated using IDW to create a smooth spatial field of normalized dimensionless ratios to accommodate for areas with no observational data. The ratio of the IDW value was limited between 0.1 to 10 to prevent the spatial interpolation of any outlier CMAQ value that was over one order of magnitude larger or smaller than the measured value. This field is then multiplied with the adjusted field of CMAQ values to create the final fused field C* (Equation (5)).
C * = ( O B S O B S ¯ · α C M A Q ¯ β α C M A Q β ) I D W × α C M A Q β
This method has fewer steps than previous data fusion methods that allows for easier implementation and less computational demands [20,21]. Equation (5) is applied daily to create a new fused field every day.

2.4. Model Performance Methods

Pearson squared (R2) was calculated temporally and spatially against the observational data to characterize method performance. The data for the temporal analysis is found by comparing the daily observational data at a monitor to the corresponding fused and CMAQ data values at the same location as the monitor. This is repeated for every monitor available for a particular pollutant in a given year. The spatial R2 is found by taking the correlation every day between the given observations on a particular day and the corresponding fused and CMAQ values (Equations (6) and (7)).
t e m p o r a l   R 2 = ( c o r r ( O B S m ( t ) ,   m o d e l e d m ( t ) ) 2
s p a t i a l   R 2 = ( c o r r ( O B S t ,   m o d e l e d t ) ) 2
Here m represents the monitor and t represents the day.

2.5. Spatial Plots

To visualize the concentrations spatially, each pollutant is plotted geospatially using the CMAQ cell centroid coordinate information (latitude/longitude). Each grid cell is averaged over the ten-year time span and normalized to the maximum average concentration during the ten-year time frame (Equations (8) and (9)).
C n o r m * = C * ¯ C m a x * ¯
C * ¯ = 1 n i = 1 n C *
Here C* represents the fused concentration, the overbar represents the average over the ten-year time period, and n represents the total number of days in the domain.

2.6. Model Evaluation Methods

Random data withholding is used to evaluate the performance of the DF method. Ten groups of observational data are created, each with a unique 10% of the total monitoring data. The monitor data withheld in each set was chosen at random to eliminate spatial biases. The above data fusion method is performed with the remaining 90% of data for each of the ten groups, resulting in ten new fused fields. We then did our model performance measures on the new field at the removed measurement locations of that particulate group. All reported withholding results are the average of each of the ten runs. Ten runs were chosen to ensure the stability of the evaluation. The spatiotemporal R2 and normalized root mean squared error (NRMSE) were calculated at withheld data points to evaluate the model (Equations (10) and (11)).
R 2 = ( c o r r ( O B S ,   C * ) ) 2
N R M S E = 1 N i = 1 N ( C i *     O B S i ) 2 O B S ¯
The OBS term represents all withheld observation points, C* represents the fused field at those corresponding observation points, N represents the total number of withheld points, and the overbar represents the averaging of all observation points.

2.7. Population Weighted Average Exposure

Population weighted average exposure was calculated using the results from the fused fields over the ten-year domain for all species to understand long-term trends in air pollution. A fractional gridded population was calculated by determining the population in each 12 km CMAQ grid cell (Equation (12)).
F p o p = P g r i d P T
Here Fpop represents the fractional value of population in a grid cell, Pgrid represents the population in the grid cell, and PT represents the total population in the contiguous United States. The population data are taken from the publicly available US Census Bureau. The 2010 census data was used for each year [43]. The tract data contains each tract’s population, the tract area, and coordinates for the population centroid [43]. Population in each cell was estimated by assigning cells to tracts and dividing the population of the tract equally among all of its assigned CMAQ grid cells. A cell was assigned to a tract if the centroid of a given cell fell within that tract. The fractional grid level population is then multiplied to the fused concentration and summed over the contiguous United States to determine the population weighted averaged exposure (Equation (13)).
P o p u l a t i o n   W e i g h t e d   E x p o s u r e =   i = 1 n ( F p o p ,   i * C i * )
The subscript i references the individual grid cell, n represents the total number of grid cells over the contiguous United States, and C* represents the calculated fused concentration.

3. Results

Four sets of results are presented. The first are normalized dimensionless plots of the resulting fused fields. Second, is the comparison of the performance metrics and the improvements made to the fused fields from the initial CMAQ fields. Third, the model evaluation results from the 10% data withholding and finally the results from the population weighted exposure.
Figure 2 The normalized fused fields averaged over the ten-year time span, normalized according to Equation (8).
The fused normalized fields varied across the 12 pollutants, which is expected because of the different nature of each species. Mobile source pollutants such as NOx, NO2, and CO were found to highlight metropolitan areas. Ozone was found to have the highest normalized values with high simulated concentrations over the Atlantic Ocean and Gulf of Mexico. SO2, a primary pollutant, was found to have a less homogeneous spatial distribution with high concentrations in the eastern contiguous United States area. Particulate species were found to have higher values in the eastern contiguous United States compared to the western side.
Model performance (Figure 3) was found to vary between species (Equations (6) and (7)). Pollutant estimates show an improved temporal and spatial R2 over the initial CMAQ fields to varying degrees. The NRMSE was highest in species that produced the most scatter in the initial CMAQ and observational data regression, such as PM10 and SO2 (Table S2). The scatter is quantified in Table S1 where the R2 of the annual average CMAQ and observational regression is calculated.
All species showed an increase in performance for all metrics compared to the initial CMAQ results. O3, PM10, and SO2 were found to have the greatest increase in performance temporally, with SO2 having the lowest CMAQ and fused R2. These species (PM10 and SO2) had an increase in temporal R2 of 0.70. Temporal performance metrics were found to be higher than spatial metrics for all 12 species. The highest temporal R2 was seen for ozone at 0.98, with the lowest for SO2 at 0.84. Spatially, SO42− performed the best at 0.91, with SO2 as the lowest with 0.42. The greatest increase in spatial performance occurred for OC with an improved spatial R2 of 0.48.
Temporal performance was further examined by comparing the fused field to the observations on the different types of monitor sampling days. This is an additional way to quantify the effects of the different number of temporal observations available throughout the year. This data fusion method does not change based on the sampling frequency of the monitor, and accounts for monitors with both high and low sampling frequencies. Three pollutants are highlighted for this analysis: EC, SO42−, and PM2.5. EC was chosen to highlight the effects of a sparsely populated monitoring network, SO42− was chosen to show a secondary pollutant, and PM2.5 was shown to highlight a highly populated monitoring network. Speciated PM monitors have two sampling frequencies: 1 in 3 days and 1 in 6 days. PM2.5 had the same two sampling frequencies, with additional daily sampling monitors. Table 3 compares the R2 and NRMSE on the different sampling frequencies. Day A represents the one in three sampling frequency, Day B represents the 1 in 6 sampling frequency, and Day C represents the daily sampling frequency for PM2.5. For speciated PM, Day A would have the fewest observational points because only the 1 in 3 day monitors are reporting a value. Day B would have the most observational points because both the 1 in 3 day monitors and 1 in 6 day monitors are reporting a value. For PM2.5 Day C would have the fewest observational points because only the daily monitors are reporting a value.
For the ten years, EC on average had an increased R2 of 0.02 and a decreased NRMSE of 0.02 on Day B compared to Day A. SO42− on average was not found to have a difference between Day A and Day B for R2 and NRMSE. PM2.5 was found to have decreased R2 and increased NRMSE for Day C compared to A and B. For PM2.5 and EC we see that performance is enhanced on days with more observational data.
A model evaluation was done by performing a 10% data withholding and rerunning the model to predict the fused concentration at the missing data points. The R2 and NRMSE for the data withholding are shown in Figure 4.
Ozone had the best evaluation metrics with the highest R2 of 0.86 and NRMSE of 0.14. All species had R2 and NRMSE with an improved withholding value compared to the original CMAQ model results. SO2 performed the worst with the lowest R2 of 0.21 and an NRMSE of 1.03. The R2 was found to increase for all species compared to the original modeled results on average of 0.22, with PM2.5 having the greatest increase of 0.35. The NRMSE was found to decrease for the withheld fields on an average of 0.14. NO2 was found to have the largest decrease of 0.36.
The evaluation metrics were split up between the western and eastern contiguous United States to further evaluate model spatial performance. Table 4 gives the specific eastern and western evaluation R2 and NRMSE for PM2.5, SO42−, and EC. The western contiguous United States is defined as a monitor which falls west of the longitude line −94.6046. The metrics shown are the average of all the monitors within the respective domain for each particular year.
On average the eastern contiguous United States was found to have an increased R2 compared to the western contiguous United States of 0.14 and 0.18 for PM2.5 and SO42−, respectively. This difference is found to be less pronounced in more recent years (2014) compared to earlier years. EC on average had a smaller difference in R2 of 0.02. PM2.5 and SO42− had an average decrease in NRMSE of 0.15 and 0.10, respectively, with EC having an average decrease in NRMSE of 0.08.
The fused results were then used to calculate a population weighted concentration across the contiguous United States (Equation (13)) and shown in Figure 5. This population weighted concentration was calculated for each CONUS covering grid cell and averaged nationally to determine an averaged contiguous United States population weighted exposure.
Pollutants such as SO2 and NOx were found to have the largest decrease rates in concentration with a slope of −4.8 × 10−3 and −1.8 × 10−3, respectively (Table S4). The slopes were calculated using linear regression for each species. All pollutants show a decrease in concentration for the time period as seen with the negative slopes (Table S4). The oscillating peaks and valleys represent seasonal trends, which are more pronounced for species like O3 that have known seasonal patterns, and less pronounced for species like OC and NH4+ [10].

4. Discussion

The results from this data fusion method have the advantage of keeping the original spatial patterns and completeness of the CMAQ results, while also matching the temporal variability and correlation to the monitoring data. Yearly linear regression between the annual averaged CMAQ and observations (Equation (1)) was found to increase the variability regression parameters each year, in comparison to power regression. IDW was chosen over other spatial interpolation methods, such as ordinary kriging, because it provided greater stability with our daily spatial interpolations and was computationally less demanding.
Although the annual mean model results in Figure S1 show scatter between the original CMAQ and observational data, a perfect 1:1 relationship is also not encouraged. Simulation CTM concentrations are an average concentration over a 12 km × 12 km grid, while the observational data represents a point measurement. Due to the wide nature of what falls within the CMAQ grid cell (highways, buildings) we should not expect the CMAQ to identically match the point source observational data. The goal of the annual mean model is not to match the CMAQ to the observational data identically, rather to capture the overarching trend of the observational data.
This method increased the temporal and spatial performance for all 12 species compared to the initial CMAQ field in both the performance and the evaluation. For the purposes of epidemiology and health studies temporal accuracy is important, making these results useful for these purposes. In comparison to solely using observational data, this method provides information in locations where observational data are sparse. This is particularly an issue for speciated particulate matter such as EC, OC, NO3, NH4+, and SO42− and for all species in rural areas.
A second advantage of this data fusion method is the scalability over a large domain (contiguous United States) and a wide range of species, without the need for calculating additional spatial and seasonal correction parameters. The ability to apply this method over the contiguous United States makes it computationally efficient, and easy to apply to other study domains, CTM field resolutions, and species. Current fusion methods with satellite data are limited primarily to gases and PM2.5 and do not deal with speciated PM on such a large domain. Observational data are available for a wider range of species compared to satellite data, allowing this method to improve model predictions for a wider range of species.
Furthermore, performing this fusion method over a ten-year domain allows us to visualize and quantify the rate of change in a species concentration over time (Figure 5). Decreases seen in mobile source pollutants such as CO, NO2, and NOX reflect the improved car engine performance, control strategies, and increased regulation (Table S4). The decrease seen in SO2 reflects the control strategies used at coal fired power plants (Table S4). However, this method of data visualization is limited by the assumption of equally-distributed population weighted exposure within a grid cell.
This method can be limited by CMAQ’s performance in modeling certain species. Table 2 demonstrates that CMAQ has varying levels of performance in predicting different species, as indicated by the differing levels of R2 in the annual mean regression between the CMAQ and observations. For species such as PM10 with a lower R2 of 0.03–0.14, this can be due to limitations in understanding certain atmospheric and physical properties related to PM10. When CMAQ has limitations in modeling a species, the fusion method will also be limited in the improvements it can make. This can be seen in PM10 by the fused R2 being lower than other species like SO42−, which CMAQ originally predicts well (R2 of 0.77–0.93).
A second potential limitation comes from the availability of monitoring data [10,11,12,15]. Although monitoring data are accurate, the spatial availability for speciated PM is lower compared to gases. In Table 1 we see than the monitors available for the speciated PM is an order of magnitude less than those available for gases such as ozone. This is evident in the spatial R2 performance. Every species showed an improvement of at least 50% from the original CMAQ data, however those with fewer monitors, EC and OC, showed a lesser improvement compared to species with a higher number of monitors (PM10, PM2.5, O3).
A monitoring network not included in this study was the Interagency Monitoring of Protected Visual Environments (IMPROVE) network which includes speciated PM2.5 concentrations in class 1 visibility areas throughout the contiguous United States [44]. The greatest improvement in including this dataset would likely occur in rural locations which previously had sparse observational data. Specifically, improvements would be greatest for species such as EC and OC in the western contiguous United States, where the Chemical Speciation Network reports few measurements and CMAQ does not predict the species as well as others (Table 2). There will also be an improvement to a lesser degree in the ion species as well (NO3, NH4+, SO42−). The ion species are more homogenous across the contiguous United States and are better predicted by CMAQ, compared to EC and OC. The IMPROVE network was not included in this study as an oversight, however the methodology to incorporate IMPROVE and any future observational datasets would not change.

5. Conclusions

The method described in this paper led to improved spatiotemporal exposure fields for 12 pollutants. This method is more readily implemented than previous data fusion methods. We have shown that this method is able to improve the temporal and spatial correlations compared to the initial CMAQ fields. Furthermore, this method has the potential to be applied to a wider range of species and a different domain, provided observational data are available.

Supplementary Materials

The following are available online at https://www.mdpi.com/1660-4601/16/18/3314/s1.

Section S1. Metadata for 12km CMAQv5.0.2. Simulations

Author Contributions

N.S. executed the model, investigation and writing the original draft; M.G. assisted with the data analysis and reviewing/editing the draft; F.M. helped develop the model; A.G.R. and J.A.M. were responsible for directly guiding and supervising the fusion work; H.H.C. oversaw the project as a whole. All authors were involved in the review and editing process.

Funding

This research was supported by the National Institute of Environmental Health Sciences of the National Institute of Health under award number R01-ES027892. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institute of Health.

Acknowledgments

Assistance provided by the U.S. EPA, particularly Wyat Appel and Valerie Garcia, is greatly appreciated for providing the CMAQ modeling results. We would also like to acknowledge the National Institute of Health under grant R01-ES027892 for supporting this work.

Conflicts of Interest

The authors declare no conflicts of interest. The authors declare that the research was conducted in the absence of commercial or financial relationships that could be construed as a potential conflict of interest.

References

  1. Pope, C., III. Epidemiology of fine particulate air pollution and human health: Biologic mechanisms and who’s at risk? Environ. Health Perspect. 2000, 108 (Suppl. 4), 713–723. [Google Scholar] [CrossRef] [PubMed]
  2. Goldman, G.T.; Mulholland, J.A.; Russell, A.G.; Srivastava, A.; Strickland, M.J.; Klein, M.; Waller, L.A.; Tolbert, P.E.; Edgerton, E.S. Ambient air pollutant measurement error: Characterization and impacts in a time-series epidemiologic study in Atlanta. Environ. Sci. Technol. 2010, 44, 7692–7698. [Google Scholar] [CrossRef] [PubMed]
  3. Isakov, V.; Crooks, J.L.; Touma, J.; Valari, M. Development and evaluation of alternative metrics of ambient air pollution exposure for use in epidemiologic studies. In Air Pollution Modeling and Its Application XXI; Springer: Berlin/Heidelberg, Germany, 2011; pp. 681–686. [Google Scholar]
  4. Sarnat, S.E.; Sarnat, J.A.; Mulholland, J.; Isakov, V.; Özkaynak, H.; Chang, H.H.; Klein, M.; Tolbert, P.E. Application of alternative spatiotemporal metrics of ambient air pollution exposure in a time-series epidemiological study in Atlanta. J. Expo. Sci. Environ. Epidemiol. 2013, 23, 593–605. [Google Scholar] [CrossRef] [PubMed]
  5. Bell, M.L. The use of ambient air quality modeling to estimate individual and population exposure for human health research: A case study of ozone in the Northern Georgia Region of the United States. Environ. Int. 2006, 32, 586–593. [Google Scholar] [CrossRef] [PubMed]
  6. Tolbert, P.E.; Klein, M.; Peel, J.L.; Sarnat, S.E.; Sarnat, J.A. Multipollutant modeling issues in a study of ambient air quality and emergency department visits in Atlanta. J. Expo. Sci. Environ. Epidemiol. 2007, 17, S29–S35. [Google Scholar] [CrossRef] [PubMed][Green Version]
  7. Peel, J.L.; Tolbert, P.E.; Klein, M.; Metzger, K.B.; Flanders, W.D.; Todd, K.; Mulholland, J.A.; Ryan, P.B.; Frumkin, H. Ambient air pollution and respiratory emergency department visits. Epidemiology 2005, 16, 164–174. [Google Scholar] [CrossRef] [PubMed]
  8. Baxter, L.K.; Dionisio, K.L.; Burke, J.; Sarnat, S.E.; Sarnat, J.A.; Hodas, N.; Rich, D.Q.; Turpin, B.J.; Jones, R.R.; Mannshardt, E.; et al. Exposure prediction approaches used in air pollution epidemiology studies: Key findings and future recommendations. J. Expo. Sci. Environ. Epidemiol. 2013, 23, 654–659. [Google Scholar] [CrossRef][Green Version]
  9. Marshall, J.D.; Nethery, E.; Brauer, M. Within-urban variability in ambient air pollution: Comparison of estimation methods. Atmos. Environ. 2008, 42, 1359–1369. [Google Scholar] [CrossRef]
  10. Appel, K.W.; Gilliland, A.B.; Sarwar, G.; Gilliama, R.C. Evaluation of the Community Multiscale Air Quality (CMAQ) model version 4.5: Sensitivities impacting model performance: Part I—Ozone. Atmos. Environ. 2007, 41, 9603–9615. [Google Scholar] [CrossRef]
  11. Appel, K.W.; Bhave, P.V.; Gilliland, A.B.; Sarwar, G.; Roselle, S.J. Evaluation of the community multiscale air quality (CMAQ) model version 4.5: Sensitivities impacting model performance: Part II—Particulate matter. Atmos. Environ. 2008, 42, 6057–6066. [Google Scholar] [CrossRef]
  12. Binkowski, F.S.; Roselle, S.J. Models-3 Community Multiscale Air Quality (CMAQ) model aerosol component 1. Model description. J. Geophys. Res. Space Phys. 2003, 108. [Google Scholar] [CrossRef]
  13. Hao, H.; Chang, H.H.; Holmes, H.A.; Mulholland, J.A.; Klein, M.; Darrow, L.A.; Strickland, M.J. Air pollution and preterm birth in the US State of Georgia (2002–2006): Associations with concentrations of 11 ambient air pollutants estimated by combining Community Multiscale Air Quality Model (CMAQ) simulations with stationary monitor measurements. Environ. Health Perspect. 2015, 124, 875–880. [Google Scholar] [CrossRef] [PubMed]
  14. Ivey, C.E.; Holmes, H.A.; Hu, Y.; Mulholland, J.A.; Russell, A.G. A method for quantifying bias in modeled concentrations and source impacts for secondary particulate matter. Front. Environ. Sci. Eng. 2016, 10, 14. [Google Scholar] [CrossRef]
  15. Baek, J.; Hu, Y.; Odman, M.T.; Russell, A.G. Modeling secondary organic aerosol in CMAQ using multigenerational oxidation of semi-volatile organic compounds. J. Geophys. Res. Space Phys. 2011, 116. [Google Scholar] [CrossRef][Green Version]
  16. Appel, K.W.; Chemel, C.; Roselle, S.J.; Francis, X.V.; Hu, R.-M.; Sokhi, R.S.; Rao, S.; Galmarini, S. Examination of the Community Multiscale Air Quality (CMAQ) model performance over the North American and European domains. Atmos. Environ. 2012, 53, 142–155. [Google Scholar] [CrossRef][Green Version]
  17. Canty, T.P.; Hembeck, L.; Vinciguerra, T.P.; Anderson, D.C.; Goldberg, D.L.; Carpenter, S.F.; Allen, D.J.; Loughner, C.; Salawitch, R.J.; Dickerson, R.R. Ozone and NOx chemistry in the eastern US: Evaluation of CMAQ/CB05 with satellite (OMI) data. Atmos. Chem. Phys. Discuss. 2015, 15, 4427–4461. [Google Scholar] [CrossRef]
  18. Zhang, H.; Chen, G.; Hu, J.; Chen, S.-H.; Wiedinmyer, C.; Kleeman, M.; Ying, Q. Evaluation of a seven-year air quality simulation using the Weather Research and Forecasting (WRF)/Community Multiscale Air Quality (CMAQ) models in the eastern United States. Sci. Total Environ. 2014, 473, 275–285. [Google Scholar] [CrossRef]
  19. Foley, K.M.; Roselle, S.J.; Appel, K.W.; Bhave, P.V.; Pleim, J.E.; Otte, T.L.; Mathur, R.; Sarwar, G.; Young, J.O.; Gilliam, R.C.; et al. Incremental testing of the Community Multiscale Air Quality (CMAQ) modeling system version 4.7. Geosci. Model Dev. 2010, 3, 205–226. [Google Scholar] [CrossRef][Green Version]
  20. Friberg, M.D.; Zhai, X.D.; Holmes, H.; Chang, H.H.; Strickland, M.; Sarnat, S.E.; Tolbert, P.E.; Russell, A.G.; Mulholland, J.A. Method for Fusing Observational Data and Chemical Transport Model Simulations to Estimate Spatiotemporally-Resolved Ambient Air Pollution. Environ. Sci. Technol. 2016, 50, 3695–3705. [Google Scholar] [CrossRef]
  21. Huang, R.; Zhai, X.; Ivey, C.E.; Friberg, M.D.; Hu, X.; Liu, Y.; Di, Q.; Schwartz, J.; Mulholland, J.A.; Russell, A.G. Air pollutant exposure field modeling using air quality model-data fusion methods and comparison with satellite AOD-derived fields: Application over North Carolina, USA. Air Qual. Atmos. Health 2018, 11, 11–22. [Google Scholar] [CrossRef]
  22. Zhai, X.; Russell, A.G.; Sampath, P.; Mulholland, J.A.; Kim, B.-U.; Kim, Y.; D’Onofrio, D. Calibrating R-LINE model results with observational data to develop annual mobile source air pollutant fields at fine spatial resolution: Application in Atlanta. Atmos. Environ. 2016, 147, 446–457. [Google Scholar] [CrossRef][Green Version]
  23. Shaddick, G.; Thomas, M.L.; Green, A.; Brauer, M.; Burnett, A.v.; Chang, H.H.; Cohen, A.; van Dingenen, R.; Dora, C. Data integration model for air quality: A hierarchical approach to the global estimation of exposures to ambient air pollution. J. R. Stat. Soc. Ser. C Appl. Stat. 2018, 67, 231–253. [Google Scholar] [CrossRef]
  24. Wang, Y.; Hu, X.; Chang, H.H.; Waller, L.A.; Belle, J.H.; Liu, Y. A Bayesian Downscaler Model to Estimate Daily PM2.5 Levels in the Conterminous US. Int. J. Environ. Res. Public Health 2018, 15, 1999. [Google Scholar] [CrossRef] [PubMed]
  25. Sampson, P.D.; Richards, M.; Szpiro, A.A.; Bergen, S.; Sheppard, L.; Larson, T.V.; Kaufman, J.D. A regionalized national universal kriging model using Partial Least Squares regression for estimating annual PM2.5 concentrations in epidemiology. Atmos. Environ. 2013, 75, 383–392. [Google Scholar] [CrossRef] [PubMed][Green Version]
  26. Zhan, Y.; Luo, Y.; Deng, X.; Chen, H.; Grieneisen, M.L.; Shen, X.; Zhu, L.; Zhang, M. Spatiotemporal prediction of continuous daily PM 2.5 concentrations across China using a spatially explicit machine learning algorithm. Atmos. Environ. 2017, 155, 129–139. [Google Scholar] [CrossRef]
  27. Birant, D. Comparison of Decision Tree Algorithms for Predicting Potential Air Pollutant Emissions with Data Mining Models. J. Environ. Inform. 2011, 17, 46–53. [Google Scholar] [CrossRef]
  28. Yu, H.; Russell, A.; Mulholland, J.; Odman, T.; Hu, Y.; Chang, H.H.; Kumar, N.; Mullholland, J. Cross-comparison and evaluation of air pollution field estimation methods. Atmos. Environ. 2018, 179, 49–60. [Google Scholar] [CrossRef]
  29. Berrocal, V.J.; Gelfand, A.E.; Holland, D.M. Space-time data fusion under error in computer model output: An application to modeling air quality. Biometrics 2012, 68, 837–848. [Google Scholar] [CrossRef]
  30. Hu, X.; Belle, J.H.; Meng, X.; Wildani, A.; Waller, L.A.; Strickland, M.J.; Liu, Y. Estimating PM2.5 Concentrations in the Conterminous United States Using the Random Forest Approach. Environ. Sci. Technol. 2017, 51, 6936–6944. [Google Scholar] [CrossRef]
  31. Jinnagara Puttaswamy, S.; Nguyen, H.M.; Braverman, A.; Hu, X.; Liu, Y. Statistical data fusion of multi-sensor AOD over the continental United States. Geocarto Int. 2014, 29, 48–64. [Google Scholar] [CrossRef]
  32. Di, Q.; Kloog, I.; Koutrakis, P.; Lyapustin, A.; Wang, Y.; Schwartz, J. Assessing PM2.5 Exposures with High Spatiotemporal Resolution across the Continental United States. Environ. Sci. Technol. 2016, 50, 4712–4721. [Google Scholar] [CrossRef] [PubMed]
  33. Al-Saadi, J.; Szykman, J.; Pierce, R.B.; Kittaka, C.; Neil, D.; Chu, D.A.; Gumley, L.; Prins, E.; Weinstock, L.; Macdonald, C.; et al. Improving National Air Quality Forecasts with Satellite Aerosol Observations. Bull. Am. Meteorol. Soc. 2005, 86, 1249–1262. [Google Scholar] [CrossRef]
  34. Xue, T.; Zheng, Y.; Geng, G.; Zheng, B.; Jiang, X.; Zhang, Q.; He, K. Fusing Observational, Satellite Remote Sensing and Air Quality Model Simulated Data to Estimate Spatiotemporal Variations of PM2.5 Exposure in China. Remote Sens. 2017, 9, 221. [Google Scholar] [CrossRef]
  35. Friberg, M.D.; Kahn, R.A.; Holmes, H.A.; Chang, H.H.; Sarnat, S.E.; Tolbert, P.E.; Russell, A.G.; Mulholland, J.A. Daily ambient air pollution metrics for five cities: Evaluation of data-fusion-based estimates and uncertainties. Atmos. Environ. 2017, 158, 36–50. [Google Scholar] [CrossRef]
  36. Fuentes, M.; Song, H.-R.; Ghosh, S.K.; Holland, D.M.; Davis, J.M. Spatial Association between Speciated Fine Particles and Mortality. Biometrics 2006, 62, 855–863. [Google Scholar] [CrossRef] [PubMed]
  37. Zhang, Y.; Foley, K.M.; Schwede, D.B.; Bash, J.O.; Pinto, J.P.; Dennis, R.L. A Measurement-Model Fusion Approach for Improved Wet Deposition Maps and Trends. J. Geophys. Res. Atmos. 2019, 124, 4237–4251. [Google Scholar] [CrossRef]
  38. US EPA Office of Research and Development. CMAQv5.0.2. Zenodo. 30 May 2014. Available online: http://doi.org/10.5281/zenodo.1079898 (accessed on 3 August 2017).
  39. US EPA. National Emissions Inventory (NEI), Facility-Level, US, 2005, 2008, 2011. Available online: https://www.epa.gov/air-emissions-inventories/national-emissions-inventory (accessed on 10 August 2019).
  40. US EPA. AirData Download Files Documentation. 2015. Available online: https://aqs.epa.gov/aqsweb/airdata/FileFormats.html (accessed on 3 August 2017).
  41. US EPA. NAAQS Table. Available online: https://www.epa.gov/criteria-air-pollutants/naaqs-table (accessed on 27 March 2019).
  42. US EPA. Air Quality System Database. Available online: http://www.epa.gov/ttn/airs/aqsdatamaty (accessed on 3 August 2017).
  43. Wong, D.W.; Yuan, L.; Perlin, S.A. Comparison of spatial interpolation methods for the estimation of air quality data. J. Expo. Sci. Environ. Epidemiol. 2004, 14, 404. [Google Scholar] [CrossRef]
  44. U.S. Census Bureau. Decennial Census of Population and Housing. Available online: https://www.census.gov/programs-surveys/decennial-census/data/tables.2010.html (accessed on 20 February 2019).
Figure 1. Comparison of monitor coverage between particulate matter (PM2.5) and elemental carbon (EC) for the year 2011. The different colored dots represent the different temporal sampling frequencies present for particulate species.
Figure 1. Comparison of monitor coverage between particulate matter (PM2.5) and elemental carbon (EC) for the year 2011. The different colored dots represent the different temporal sampling frequencies present for particulate species.
Ijerph 16 03314 g001
Figure 2. Normalized concentrations for the 12 species averaged over the ten-year time span. The concentrations are normalized against the maximum value over the contiguous United States (CONUS), excluding Mexico and Canada. The values plotted are dimensionless.
Figure 2. Normalized concentrations for the 12 species averaged over the ten-year time span. The concentrations are normalized against the maximum value over the contiguous United States (CONUS), excluding Mexico and Canada. The values plotted are dimensionless.
Ijerph 16 03314 g002
Figure 3. Comparison of fused field metrics (Pearson R2) to the original CMAQ model. The blue bar represents the fused field metric, while the black line for each species represents the CMAQ metric.
Figure 3. Comparison of fused field metrics (Pearson R2) to the original CMAQ model. The blue bar represents the fused field metric, while the black line for each species represents the CMAQ metric.
Ijerph 16 03314 g003
Figure 4. R2 and NRMSE used to evaluate model performance on the withheld dataset. The plots represent the average over each of the 10 withholding runs that were performed. The black line for each species represents the metric for the raw CMAQ model, while the blue bar represents the calculated withheld metric of the fused field.
Figure 4. R2 and NRMSE used to evaluate model performance on the withheld dataset. The plots represent the average over each of the 10 withholding runs that were performed. The black line for each species represents the metric for the raw CMAQ model, while the blue bar represents the calculated withheld metric of the fused field.
Ijerph 16 03314 g004
Figure 5. Nationwide averaged population weighted ambient concentrations plotted over the ten-year domain of 2005–2014 for the twelve pollutants. The y-axis shows the population weighted concentration with the units shown in the title, and the x-axis sows the dates ranging from 1 January 2005–31 December 2014.
Figure 5. Nationwide averaged population weighted ambient concentrations plotted over the ten-year domain of 2005–2014 for the twelve pollutants. The y-axis shows the population weighted concentration with the units shown in the title, and the x-axis sows the dates ranging from 1 January 2005–31 December 2014.
Ijerph 16 03314 g005
Table 1. Summary statistics for the observational data from 2005–2014. A range is presented to show the changes in observational data over the ten-year span. Completeness is defined as the percentage of days during the year with measurement data available. Monitor total represents the total number of active monitors during the year, while daily represents the number of active monitors that take daily measurements during the year. Total observations (OBS) represents the total measurements taken during the ten-year span.
Table 1. Summary statistics for the observational data from 2005–2014. A range is presented to show the changes in observational data over the ten-year span. Completeness is defined as the percentage of days during the year with measurement data available. Monitor total represents the total number of active monitors during the year, while daily represents the number of active monitors that take daily measurements during the year. Total observations (OBS) represents the total measurements taken during the ten-year span.
PollutantMonitor TotalDailyTotal OBSCompleteness (%)
Particulate SpeciesPM10691–999248–3201,416,22637–55
PM2.5768–1071114–1831,231,79535–39
EC95–172087,77613–22
OC95–172085,73413–22
SO42−103–1720102,31120–23
NO3103–172090,17620–23
NH4+102–172094,33219–22
GasesNOx308–419268–3711,164,91285–90
NO2300–400270–3891,373,56987–90
CO303–418278–3931,257,73487–90
O31182–1265556–7403,439,16975–80
SO2429–507396–4811,572,60190–93
Table 2. The calculated parameter range for the α and β parameters and the range of the R2 for the fit between the community multiscale air quality (CMAQ) and observations. Each range encompasses the values used during the ten-year period.
Table 2. The calculated parameter range for the α and β parameters and the range of the R2 for the fit between the community multiscale air quality (CMAQ) and observations. Each range encompasses the values used during the ten-year period.
α β R2
Particulate SpeciesPM1011.67–15.640.10–0.230.03–0.14
PM2.53.68–4.620.32–0.500.28–0.50
EC0.58–0.840.34–0.540.26–0.49
OC1.45–1.800.18–0.450.10–0.35
SO42−1.11–1.310.82–1.010.77–0.93
NO30.88–1.290.53–0.890.43–0.60
NH4+0.74–1.450.46–0.760.24–0.78
GasesNOx2.04–4.200.62–0.940.48–0.65
NO21.66–2.210.68–0.760.63–0.71
CO0.82–1.090.43–0.670.16–0.36
O30.32–0.730.64–0.910.50–0.65
SO21.36–3.550.69–0.950.37–0.54
Table 3. Comparison of the R2 and normalized root mean squared error (NRMSE) between C* and observations on the different sampling days. Speciated PM monitors, EC and SO42− have two sampling frequencies, and PM2.5 has three sampling frequencies. Day A represents 1 in 3 days, Day B represents 1 in 6 days, and Day C represents daily measurements (only for PM2.5). The metrics shown are the averages of all days included in each of the three categories.
Table 3. Comparison of the R2 and normalized root mean squared error (NRMSE) between C* and observations on the different sampling days. Speciated PM monitors, EC and SO42− have two sampling frequencies, and PM2.5 has three sampling frequencies. Day A represents 1 in 3 days, Day B represents 1 in 6 days, and Day C represents daily measurements (only for PM2.5). The metrics shown are the averages of all days included in each of the three categories.
20052006200720082009
R2NRMSER2NRMSER2NRMSER2NRMSER2NRMSE
EC1 in 30.570.520.610.440.610.480.640.450.760.31
1 in 60.590.490.620.420.640.440.680.380.760.31
SO42−1 in 30.900.240.910.520.920.180.870.230.920.20
1 in 60.900.250.900.540.930.180.890.230.910.21
PM2.51 in 30.810.230.800.230.820.230.810.220.810.22
1 in 60.810.240.800.240.810.230.810.220.800.22
daily0.770.260.760.250.800.250.800.270.800.30
20102011201220132014
R2NRMSER2NRMSER2NRMSER2NRMSER2NRMSE
EC1 in 30.700.380.710.340.690.370.760.340.690.36
1 in 60.680.410.720.320.720.310.760.330.710.33
SO42−1 in 30.850.310.910.190.900.210.890.200.870.24
1 in 60.840.300.910.180.890.210.900.200.880.24
PM2.51 in 30.790.230.810.220.740.250.770.260.730.27
1 in 60.780.230.810.220.740.260.760.260.740.28
daily0.780.240.780.240.740.260.750.260.720.27
Table 4. Model evaluation metrics, R2 and NRMSE divided spatially into the eastern and western contiguous United States for PM2.5, SO42−, and EC. The metrics are shown for each year and are the average of the monitors in the domain.
Table 4. Model evaluation metrics, R2 and NRMSE divided spatially into the eastern and western contiguous United States for PM2.5, SO42−, and EC. The metrics are shown for each year and are the average of the monitors in the domain.
20052006200720082009
R2NRMSER2NRMSER2NRMSER2NRMSER2NRMSE
ECEastern0.510.420.560.400.380.520.450.470.670.36
Western0.470.540.480.510.430.590.370.580.620.43
SO42−Eastern0.810.320.760.320.790.310.740.330.710.31
Western0.520.450.570.410.540.440.480.430.560.41
PM2.5Eastern0.750.240.760.230.820.230.800.230.790.23
Western0.770.360.560.480.650.370.640.360.630.36
20102011201220132014
R2NRMSER2NRMSER2NRMSER2NRMSER2NRMSE
ECEastern0.640.400.590.370.570.390.590.390.530.40
Western0.590.450.560.480.580.450.630.440.610.44
SO42−Eastern0.680.350.730.330.650.320.720.320.650.34
Western0.420.510.580.400.580.370.610.390.540.46
PM2.5Eastern0.800.230.780.240.750.230.790.230.760.24
Western0.600.380.650.370.630.370.640.380.600.41

Share and Cite

MDPI and ACS Style

Senthilkumar, N.; Gilfether, M.; Metcalf, F.; Russell, A.G.; Mulholland, J.A.; Chang, H.H. Application of a Fusion Method for Gas and Particle Air Pollutants between Observational Data and Chemical Transport Model Simulations Over the Contiguous United States for 2005–2014. Int. J. Environ. Res. Public Health 2019, 16, 3314. https://doi.org/10.3390/ijerph16183314

AMA Style

Senthilkumar N, Gilfether M, Metcalf F, Russell AG, Mulholland JA, Chang HH. Application of a Fusion Method for Gas and Particle Air Pollutants between Observational Data and Chemical Transport Model Simulations Over the Contiguous United States for 2005–2014. International Journal of Environmental Research and Public Health. 2019; 16(18):3314. https://doi.org/10.3390/ijerph16183314

Chicago/Turabian Style

Senthilkumar, Niru, Mark Gilfether, Francesca Metcalf, Armistead G. Russell, James A. Mulholland, and Howard H. Chang. 2019. "Application of a Fusion Method for Gas and Particle Air Pollutants between Observational Data and Chemical Transport Model Simulations Over the Contiguous United States for 2005–2014" International Journal of Environmental Research and Public Health 16, no. 18: 3314. https://doi.org/10.3390/ijerph16183314

Note that from the first issue of 2016, MDPI journals use article numbers instead of page numbers. See further details here.

Article Metrics

Back to TopTop