4.1. Model and Imputation Technique Selection
Table 8 contains the RMSE for the selected models, and
Table A8,
Table A9,
Table A10 and
Table A11 contain additional metrics for said models. In terms of
,
shows the worst performance (on average, 0.8973) whereas
shows the worst RMSE (on average,
) due to the greater variability of this pollutant. Note that since the precision of all the input parameters is one unit, arguably, an error of less than one should not be relevant. A real inference scenario is illustrated in
Figure 1 and
Figure 2, which also show the location of the monitoring stations and the area of study.
We have observed that selecting the optimal model and imputation technique is a choice that depends on the time lag, the pollutant being predicted, and the city where it is being applied. Out of all the models considered in the 48 cases shown in
Table A1,
Table A2,
Table A3,
Table A4,
Table A5 and
Table A6, LSTM was selected 3 times, GRU×2 13 times, LSTM×2 18 times, LSTMGRU 12 times, and BiLSTM twice. Despite CNN’s great ability to recognize patterns, its performance is not as good as models specifically designed for time series. Likewise, neither version of MCNNLSTM was the best option for any of the analyzed cases, and GRU, RF, and LR were possibly too simple for the problem at hand, as we can see a clear dominance of models with two recursive layers. Interestingly, the dominance of LSTM×2 for 1-hour forecasting is notable, whereas for longer time lags, GRU×2 and LSTMGRU offer comparable results. For Cali predictions, there is no significant difference between these three models with two recursive layers, whereas in Madrid, we see a dominance of the LSTM×2 model. Baseline models (RF and LR) fail to offer meaningful predictions, especially for Madrid, which has a smaller dataset. They lead to a 72.5% and 203.1% increase in RMSE compared to the optimal choice, respectively. For Cali, they perform better, but still undoubtedly worse that the DL architectures proposed.
The comparison of model architectures can be interpreted as an ablation study on temporal depth and architectural composition. Single-layer recurrent models (GRU, LSTM) capture short-term dependencies, whereas two-layer recurrent architectures (GRU×2, LSTM×2, LSTMGRU) provide a hierarchical temporal representation that consistently yields better performance across pollutants and forecasting horizons, indicating the presence of multi-scale temporal dynamics in air quality time series. In contrast, although the multipath models combine CNN and LSTM branches and can be viewed as an ablation of complementary feature extractors (capturing local temporal patterns and sequential dependencies) their joint representation does not translate into superior performance, as they are never selected as the best-performing models. This result is consistent with the observed behavior of the standalone CNN models, which also fail to outperform recurrent architectures in this framework. This suggests that the extraction of temporal dependencies is more effectively handled by recurrent structures than by convolutional layers in this setting. Overall, these findings indicate that increasing temporal depth within recurrent networks is more beneficial than combining heterogeneous branches, highlighting the dominant role of hierarchical temporal modeling over architectural diversity for this task.
Regarding the imputation technique, our results show that in most cases the optimal strategy is to discard missing values rather than indiscriminately replacing them with the mean or median. This finding indicates that data quality is more critical than data quantity: a smaller but more reliable dataset can yield better predictive performance than a larger dataset contaminated by poorly imputed values. Consequently, traditional imputation strategies that do not account for data availability or reliability may degrade model performance. As shown in
Table A7, the most severe degradation due to traditional imputation occurred for Cali
forecasting with a 1-day time lag, where the RMSE increased by 74.3% relative to the optimal configuration.
For Madrid (where, as discussed in
Section 3.4, the dataset was smaller but highly reliable, with 82.7% of valid records), we found that for more than half of the scenarios, the best approach was to discard the missing values or apply continuity for one hour. For Cali, with merely 30.5% of valid records, no imputation was usually the best approach, with the traditional approach performing better only once. In general, the enhancement achieved by joint model optimization and imputation technique selection is substantial: deviations from the best configuration resulted in RMSE increases ranging from 14.1% to 115.3%. On average, our optimized approach outperformed the worst DL model by 37.9% (by 203.8% if considering RF and LR) and the worst imputation strategy by 26.6% across all cities, pollutants, and time intervals, amounting to an average improvement of 48.9%.
By analyzing autocorrelation at a 24-h lag, we assess the strength of the daily periodicity for each pollutant across different topics. The different imputation strategies have little impact on variables that already exhibit a strong 24-h cycle, such as temperature and ozone, indicating that their intrinsic daily structure is robust to missing-data handling. In contrast, pollutants with weaker inherent periodicity are more sensitive to the imputation method, with CO being the most affected, showing a noticeable reduction in its 24-h autocorrelation. These results are consistent with the previously observed behavior: exhibits the most significant degradation when using the most aggressive imputation, and it is also the only pollutant that uses CO as an input feature. This suggests that the loss of temporal structure in CO propagates through the modeling pipeline and contributes to the reduced performance of forecasting.
4.2. Evaluation on Unobserved Monitoring Sites
To assess the accuracy of our model in predicting pollution levels across the city, we have evaluated the performance of our trained model compared to locations that were not used during the training process. Specifically, the levels of the four pollutants (
,
,
, and
) were inferred by the test sensors. The results are presented in
Table 9, which shows the RMSE, and in
Table A12,
Table A13,
Table A14 and
Table A15, which show the rest of the metrics. RMSE and
are graphically represented in
Figure 5 and
Figure 6. As expected, performance was worse when tested against new locations than when tested against the validation set. Nevertheless, the results are still promising, with an average RMSE of
ppb for
,
ppb for
,
for
, and
for
. The models, which were initially developed for short-term forecasting (up to one day), exhibit anomalous behavior in a few extended-period scenarios, as evidenced by negative
values in two scenarios.Negative
values indicate that, in those scenarios, the model performed worse than a constant predictor equal to the mean of the target variable. This is likely attributable to an inability to adequately capture long-term anthropogenic pollution sources, a deficiency that could be addressed by expanding the time frame under consideration and incorporating additional inputs, such as the day of the week.
This finding aligns with the conclusions drawn from prior studies, which indicate that a longer time lag does not necessarily equate with a decline in prediction accuracy. For , the optimal results are attained for the extended 1-day time lag, whereas keeps a stable performance across time lags for Cali but degrades beyond 1-day for Madrid. The behavior of and differs between the two cities, with greater precision observed for a 4-day time lag in Cali and a 10-day time lag in Madrid, and better results for 7 days and 1 hour in Madrid, respectively.
4.3. Model Analysis
To add some insight on interpretability, a feature importance analysis is included. For each trained model we computed permutation importance by randomly shuffling the values of a single input feature across the test set (without changing the time dimension) and measuring the resulting increase in RMSE. A larger RMSE growth indicates greater feature importance because perturbing that feature degrades predictive performance more strongly.
Figure 7 reports the average RMSE increase per feature for each pollutant. Results show the expected pattern for
: The pollutant’s own past values are consistently the most important predictor. However, for
, permuting
produces the largest RMSE increase, indicating a strong predictive link between
and
in our datasets. For
, temperature is the most influential feature, while for
the importance is more uniformly distributed across the considered inputs. We also observe city-specific differences: for example, CO is a key contributor to
prediction in Madrid but is comparatively less important in Cali.
These feature-importance patterns are consistent with the physical and urban context of the studied areas, although they should be interpreted as learned statistical associations rather than direct causal attribution. For , the dominant role of temperature is coherent with the photochemical production of ozone, which is strongly modulated by solar radiation and thermal conditions. For , the strong importance of the pollutant’s own recent history, together with the contribution of CO in Madrid, is compatible with the influence of combustion-related urban traffic emissions, which are a major local source in both cities and particularly relevant in Madrid. In the case of , the high relevance of suggests that the model is exploiting traffic-related co-variation, likely linked to both exhaust emissions and mechanically resuspended road dust. This interpretation is also consistent with the known source context of Madrid, where traffic dominates and episodic Saharan dust intrusions can further increase particulate concentrations, and of Cali, where particulate matter is influenced not only by mobile sources but also by seasonal regional biomass burning. The more uniform importance distribution observed for is also reasonable, as fine particulate matter often reflects the combined effect of several correlated variables rather than a single dominant predictor.
To examine the temporal contribution of past observations, we applied a SHAP analysis (Lundberg and Lee [
42]). SHAP is an interpretability method based on cooperative game theory that quantifies the contribution of each input feature to a given prediction. In our case, it was used to estimate the relative importance of past timesteps by measuring how much each temporal input contributes to the model output. We used the
model (single seed) as a representative example. The SHAP analysis used a background set of 100 samples and explained predictions for a random subset of 150 test samples; the model was re-evaluated 25 times to stabilize estimates. We averaged the absolute SHAP values per time-lag to obtain a single importance score for each past timestep.
Figure 8 presents these timestep-averaged SHAP values across forecast horizons normalized to the range
. In both cities the most recent timestep is generally the most important predictor, but temporal importance is more evenly distributed in Cali. In Madrid (which has a shorter dataset), some long-horizon models (1-, 4-, and 7-day forecasts) rely on a narrower set of recent timesteps, suggesting a reduced capacity to capture long-term temporal structure. These temporal relevance patterns suggest that the models rely on different balances of short-term persistence and distributed temporal context depending on the city and forecast horizon, which is plausible given the differences in data availability, source mixture, and atmospheric variability between Madrid and Cali. Alternatively, averaging the SHAP values across features reveals that PM10 is the most influential predictor in Cali, whereas PM2.5 and distance contribute the least. In Madrid, PM10 and CO emerge as the most important predictors, while distance remains the least relevant feature. This indicates that temporal dynamics play a more significant role in PM10 forecasting than spatial relationships. However, distance is the dominant feature for the 1-day forecasting horizon, likely because temporal variability is partially constrained by periodic behavior at short lead times, but its importance progressively declines for longer forecasting horizons.
Regarding the training speed, the average training time for each of the selected models is included in
Table 10. The number of samples used for Cali is, approximately, three times higher. However, the training takes 5.4 times longer, on average. All the experiments were conducted on a system equipped with 144 Intel
® Xeon
® Gold 5220 CPUs except for the ozone models, which additionally used a NVIDIA
® Tesla
® M10.
To offer a meaningful comparison between our model and the work of other authors, we have selected
, the most widely studied pollutant in the literature. Coincidentally, it was the pollutant that gave the worst performance in the validation set in terms of
, as evidenced by
Table A8.
Figure 9 presents a comparative overview of our results against those reported in other studies from
Section 1.1, which also address simultaneous spatial prediction and temporal forecasting for
. The comparison includes published results from Baniasadi et al. [
7], Yang et al. [
26] (averaging all the stations listed in
Table 2 of their study), Nicolis et al. [
16] (who did not report
for temporal forecasting), and Wu et al. [
25] (for whom we selected the best-performing time lag, 7 days, among their four proposed horizons). It is important to note that these results were obtained under different conditions—each study used its own dataset, city, model, and forecasting horizon—which should be considered when interpreting the comparison. Furthermore, since most studies assess model performance at sensor locations seen during training, we used our validation set results to ensure a more equitable comparison framework.
Among the reviewed studies, Baniasadi et al. [
7] demonstrated comparable performance by employing an LSTM model that was optimized with a binary chimp optimization algorithm. However, it should be noted that the model was trained using a dataset that spanned a decade, a significantly more extensive dataset than the one used in this study. The required effort and computation cost of training a model with such a large dataset may not justify the marginal gains, especially when our models achieved similar results with far fewer data. Additionally, urban environments evolve continuously, and data from a decade ago may no longer reflect current conditions. Thus, excessively long datasets may even hinder model performance by introducing outdated patterns that are no longer representative of the present-day cityscape.
For most pollutants and time lags, the best-performing model for Madrid tended to achieve higher
scores than the best-performing model for Cali. However, it should be noted that pollutants may have a seasonal variance, and some studies (Fioravanti et al. [
18]) have even developed separate models for each month. To analyze these results, we tested the same models against the data from the last two weeks of December. Results are shown in
Table 11, and additional metrics are provided in
Table A16,
Table A17 and
Table A18. Specifically, the RMSE increased an average of 25.8% for the Cali predictions and 52.4% for the Madrid predictions. This loss of precision is especially significant for
and for longer-term forecasting. Given the limited number of results, we have opted not to include the
metric, as it tends to be misleading for a small number of samples.
While these results may initially appear to support the approach of model separation by time intervals, performance significantly degrades when we are out of the training time range by up to two weeks. Although the levels of pollutants exhibit seasonal patterns, they are not periodic and vary from year to year. By separating the model by months, we not only reduce the size of the training dataset by twelve, but we also intentionally introduce overfitting into our model, which degrades its ability to generalize. Therefore, the model’s ability to make predictions for unseen pollution levels is diminished.
To analyze the temporal stability of the proposed deep learning framework, we retrained the selected models using an extended dataset, which is also publicly available in Alvarado-Alcon et al. [
33]. This dataset spans from the initial dates reported in
Table 3 up to 25 April 2026, thus covering more than one full year for both cities. We preserved the same preprocessing pipeline and station partitioning used in the previous experiments, ensuring that evaluation remains based on held-out monitoring stations. For this extended setting, samples up to 10 September 2026 were used for training and validation, while the final 7.5 months were reserved for testing, thereby enabling evaluation on both unseen locations and unseen future time periods. The results, summarized in
Table 12, show that the proposed framework remains effective when trained on longer and more diverse datasets. In many cases, particularly for shorter forecasting horizons, the models achieve lower RMSE values than in the original experiments, indicating improved stability and learning from the increased data volume. However, a degradation in performance can still be observed for longer forecasting horizons (7 and 10 days), which remain inherently more challenging due to the increased uncertainty and weaker temporal correlations at extended lead times.
To assess operational robustness under missing data, we performed an additional experiment on the extended test subset in which non-distance input values from already constructed sliding windows were randomly replaced by the mean value of the corresponding feature and timestep (or by 0.5 when all values for a given feature and timestep were removed). The results show that the framework is highly resilient to randomly distributed missingness once a valid input window can be constructed. In Madrid, averaging across pollutants and forecast horizons, the RMSE increased by only about 0.6% for each additional 10% of substituted values between 0% and 70% missingness. In Cali, the same analysis showed no meaningful average change in RMSE over that range. Even in the most affected individual case, forecasting at a 7-day horizon in Madrid, the RMSE increase remained below 30% when 60% of the non-distance inputs were replaced. These results indicate that the trained framework is robust to substantial levels of random input corruption at inference time. Nevertheless, this experiment reflects an optimistic missing-data pattern, since real sensor outages are typically structured and continuous rather than randomly distributed. In operational deployments, missingness often arises from prolonged hardware or communication failures, which may remove the observations of multiple sensors for periods longer than the 24-h sliding window. Under these conditions, spatial information is effectively lost, and forcing predictions through aggressive imputation becomes considerably less reliable. This is consistent with the imputation technique analysis, in which more aggressive strategies led to RMSE increases of up to 74.3%, far exceeding the degradation observed under randomly distributed missingness at inference time. Therefore, while the present robustness analysis supports the operational resilience of the framework to random missing values, evaluating structured and prolonged outage scenarios remains an important limitation of air quality forecasting and a relevant direction for future work.
Finally, we note that the proposed approach is amenable to real-world, online deployment because inference is computationally lightweight. In our experiments the average inference time for the complete validation set was 2.5 s for the Cali models and 1.9 s for the Madrid models. The heatmaps shown in
Figure 1 and
Figure 2 were generated at moderate spatial resolutions (Cali: 40 × 80 grid points; Madrid: 50 × 50 grid points) and each was produced in under one minute using a single seed. These timings imply that citywide heatmaps can be recomputed on an hourly cadence without prohibitive compute cost, making the system suitable for near-real-time dashboards, public alerts, or integration with traffic and urban management tools.