Short-Term Air Pollution Forecasting Using Embeddings in Neural Networks

Abstract

Air quality is a highly relevant issue for any developed economy. The high incidence of pollution levels and their impact on human health has attracted the attention of the machine-learning scientific community. We present a study using several machine-learning methods to forecast NO₂ concentration using historical pollution data and meteorological variables and apply them to the city of Erfurt, Germany. We propose modelling the time dependency using embedding variables, which enable the model to learn the implicit behaviour of traffic and offers the possibility to elaborate on local events. In addition, the model uses seven meteorological features to forecast the NO₂ concentration for the next hours. The forecasting model also uses the seasonality of the pollution levels. Our experimental study shows that promising forecasts can be achieved, especially for holidays and similar occasions which lead to shifts in usual seasonality patterns. While the MAE values of the compared models range from $4.3$ to 15, our model achieves values of $4.4$ to $7.4$ and thus outperforms the others in almost every instance. Those forecasts again can for example be used to regulate sources of pollutants such as, e.g., traffic.

1. Introduction

Air quality forecasting has become a topic of high interest for all societies since it represents a great threat to health and the climate. Air toxicity kills approximately seven million people worldwide each year, as a result of increased mortality from stroke, heart disease, chronic obstructive pulmonary disease, lung cancer, and acute respiratory infections (https://www.who.int/health-topics/air-pollution, accessed on 10 December 2022). Statistics published by the WHO (World Health Organization) show that nine out of ten people breathe air that contains high levels of pollutants, exceeding the limits of the WHO guidelines. According to the WHO, the top six air pollutants include particulate pollution, ground-level ozone, carbon monoxide, sulphur oxides, nitrogen oxides, and lead. Nitrogen dioxide (NO₂) is a gaseous atmospheric pollutant that arises mostly as a result of road traffic and other fossil fuel combustion processes. Its presence in the air helps the formation and modification of other air pollutants, such as ozone and particles, as well as acid rain. The negative effects of NO₂ on health have been extensively studied [1,2,3], and authors have shown a high correlation between certain diseases and exposure to high concentrations of NO₂ [4,5]. In this research, we focused on the short-term forecast of NO₂ in the city of Erfurt, Germany. We forecasted the NO₂ concentration for the next 24, 72 or 120 h using a deep-learning (DL) approach, which is based on neural networks and embedding layers to encode time variables. Data access was realised using the standardised OGC SensorThings API (http://docs.opengeospatial.org/is/15-078r6/15-078r6.html, accessed on 10 December 2022) (implemented by FROST https://fraunhoferiosb.github.io/FROST-Server/, accessed on 10 December 2022). Since this covers both the historical sensor data as well as the meteorological variables, all input data could be retrieved through a unified interface. Historical NO₂ values were imported from the European Environment Agency (https://datacoveeu.github.io/API4INSPIRE/datanests/ad-hoc.html, accessed on 10 December 2022), whereas the meteorological variables were obtained from Meteomatics (https://www.meteomatics.com/de/, accessed on 10 December 2022), a provider of weather data and forecasts. Our findings are embedded in the project Bauhaus.MobilityLab (https://bauhausmobilitylab.de/, accessed on 10 December 2022), which has the goal of making urban living spaces more liveable. The resulting NO₂ forecasts were used, e.g., for measures influencing the inhabitants of the living lab. A further application is to adapt public transport rates to the expected NO₂ values. In this way, one can counteract peaks in air toxicity which are dangerous to health. The remainder of this paper is organised as follows. In Section 2, we provide an introduction to the air pollution forecasting task and the current state-of-the-art methods. In Section 3, we analyse the seasonality of the data collected by the sensors in the city of Erfurt. We use graphs to show the high seasonality in our data, as well as the difference that exists between the data of different locations. In Section 4, we introduce deep neural networks, as well as the use of embedding layers to encode categorical variables. In Section 5, we explain the setup of the experimental study, including a description of the benchmark algorithms used for comparison and present and discuss the results.

2. NO₂ Emissions and Short-Term Forecasting

Our study was focused on NO₂, a polluting gas that seriously affects human health. According to international statistics [6], in the year 2019, Germany ranked 22nd in the list of countries that emit the most NO₂ per inhabitant, with 13.6 kg per year. Although this number is quite encouraging when compared to the 107.7 kg that each inhabitant of Australia generates, there is still much that can be done to reduce it. Several studies have shown that road traffic has the highest incidence of NO₂ emission [7,8,9]. Figure 1a shows the contribution made by different sectors to emissions of nitrogen oxides in 2011. As can be seen, road transport constitutes 41% of total NO₂ emissions, being the sector with the most emissions. A study carried out in [10] states that Germany’s climate footprint has improved considerably since the 1990s and that the reasons are mainly the successful reform of the European trade system for emissions, the former low price of gas, the expansion of wind and solar energy and the closure of the first coal-fired power plants. However, the incidence of traffic is still a serious problem, which has been increasing in recent years according to a further study [10].

However, as shown in [10], not all forms of transport pollute to the same level. In Figure 1b, we can see that, at more than 60%, motorised individual transport in the form of cars had the highest incidence of emissions in the transport sector during 2017. In contrast, rail transport contributed only 0.6%. It is important to mention that the fact that the highest incidence is in “street cars” gives us some hope about taking action to reduce emissions. A good example is the city of Stuttgart, Germany. The action plan “Nachhaltig mobil in Stuttgart” (sustainable mobility in Stuttgart) was approved on 18 July 2017, by the Municipal Council. The action plan outlines more than 100 individual measures in nine fields of action including local public transport, individual motorised transport, pedestrian and bicycle traffic, commercial traffic, commuter traffic, city-specific mobility, mobility in the region and public relations work, such as intermodality and networking. The mobility package is complemented by other measures of the “Alliance for Mobility and Clean Air” of the City Council. Optimising traffic flow and changes of the model split have been proven to be the most effective measures with the least negative impact. Our NO₂ forecasts were applied to two use cases of the project Bauhaus.MobilityLab which targets an enhanced quality of living in the city of Erfurt. First, to optimise the individual traffic with regards to air quality and second, to change the modal split by giving incentives to people who use public transport when there is bad air quality. For both use cases, good knowledge about the current air quality is required. This also includes a fundamental understanding of the sources and current dispersion of bad air in the city. For example, if we know that tomorrow at 17:00 we will have an excessively high NO₂ peak in the inner city, we could try to incorporate this information into route planning algorithms, which then again could prioritise park and ride solutions. These use cases can become blueprints for urban planning in further cities.

Using Machine-Learning to Predict Pollutant Concentrations

The use of machine-learning (ML) has become popular as a powerful tool for accurate forecasts [11,12,13]. In the following, we carry out a review of the most significant methods within the state-of-the-art that make short-term forecasts of pollutants in the air. The short-term forecasting of NO₂ concentration has attracted the attention of the scientific community [14,15]. Statistical models have been used widely for prediction tasks. However, recently, ML methods have begun to be used more often [14].

In

Table 1. State-of-the-art literature overview.

Paper	Algorithm	Horizon	Compared	Data
[16]	Hybrid statistical	24 h	-	Irish EPA
[11]	LSTM encoder-decoder	5 h, 10 h, 120 h	LSTM, sequence-to-scalar	Beijing
[12]	1D CNN-GRU ${}^a$	1 h	SVR ${}^b$, DTR ${}^c$, LSTM, BGRU ${}^d$	UCI-repo
[17]	Prophet	1 h	Box-Jenkin	Bhubaneshwar
[14]	ANN e	-	BT ${}^f$, LSVM ${}^g$	Belisario, Cotocollao
[13]	Adaboost	1 h, 8 h, 24 h	SVM, ANN, Random Forest	Taiwan EPA
[18]	LGBM	-	XGB ${}^h$, LGBM	South Korea
[19]	LSTM	1 h	XGB, LGBM, LSTM	South Korea
[20]	Random Forest	-	BRT ${}^i$, SVM, XGB, GAM ${}^j$, Cubist	Hong Kong
[21]	BC-LSTM ${}^k$	6 h	C-LSTM, LSTM	Madrid
[22]	IMDA-VAE ${}^l$	-	GRU, BGRU, LSTM, VAE, C-LSTM, B-LSTM	Arizona, California, Pennsylvania, Texas
[23]	DLSTM ${}^m$	48 h	LSTM	China

^a 1-D deep convolutional gated recurrent neural network, ^b support vector regression, ^c dynamic treatment regime, ^d bidirectional GRU, ^e Artificial neural networks, ^f boosted trees, ^g linear support vector machine, ^h extreme gradient boosting, ⁱ boosted regression trees, ^j generalized additive model, ^k bidirectional convolutional LSTM, ^l integrated multiple directed attention variational autoencoder, ^m developed LSTM.

, we show some important features about the most representative methods from the state-of-the-art. As can be observed, long short-term memory (LSTM) and light gradient-boosting machine (LGBM) have been widely used in comparative studies. Furthermore, most of the models focus only on predicting NO₂ concentrations one hour in advance, which is not functional in our case study. Only four of them, one statistical model and three based on ML, predict 24-h time horizons. Therefore, we used three of those methods in our experimental study. Incorporating other methods might be part of future work.

3. NO₂ Seasonality Analysis in the City of Erfurt

The analysis of the NO₂ concentration was based on four air quality sensors distributed in the city of Erfurt (Figure 2). This data are publicly available from the European Environment Agency (EEA).

In the following we will refer to them by their official names, neglecting the prefix STA.DE:

STA.DE_DETH117	STA.DE_DETH043
STA.DE_DETH020	STA.DE_DETH081

Traffic and land use have a high impact on the NO₂ concentration [24,25,26]. There are two main factors related to traffic, influencing the NO₂ concentration. First, is the location itself. Since most of the NO₂ emission is caused by traffic, the distance to main roads and air circulation (dense buildings or rural areas) is a crucial aspect. Second, the air quality is influenced by the amount of traffic. While the location of a sensor is constant, there is a high variation in the traffic over time, due to commuter traffic and working hours or special events.

As can be seen in Figure 2, sensor DETH117 is quite far from the city, so it was expected that it will be the least affected by traffic. Sensor DETH020 is the next farthest from the centre and the busiest highways. On the other hand, sensors DETH081 and DETH043 are located in the city and near important highways. In the case of sensor DETH081, it is located very close to a major highway, the K35 (highlighted in red in Figure 2), so we expected its NO₂ concentration measurements to be high. Next, we analysed in detail the data of each sensor, in order to know their characteristics and thus suggest possible ML models that allow accurate forecasts to be made. We analysed the historical NO₂ data distribution per sensor in the first step.

The other sensors DETH043 and DETH081 behave in a very similar way, as can be seen in Figure 3. Both have almost the same IQR and average, as well as maximum and minimum values, although the maximum of sensor DETH081 is slightly higher than that of sensor DETH043. These two sensors are the closest to the city (most affected by traffic) and this is evident in the metrics shown by the box plots.

Next, we analysed the relationship between the days of the week and the NO₂ level, as well as with holidays. The objective of this analysis was to show the high seasonality of our data and the relationship that exists between the days of the week (including holidays) and the levels of NO₂. The study was divided by sensors because each of them is located in a different area and therefore the measurements may be conditioned by different factors.

In Figure 4, we have plotted the NO₂ concentrations for the four sensors during the first two weeks of September 2020. For the urban sensors (close to the city) we have used similar colours red and magenta, while for the rural sensors we have used the colours green and blue. As indicated before, the sensors located closer to the urban area show significantly higher concentrations of NO₂ than the sensors in rural areas. In Figure 5 and Figure 6, we can observe the differences in details.

The high seasonality of the concentrations of NO₂ is a proven fact, as can be seen in Figure 7 and as some authors have previously shown [27]. On weekdays, much higher concentrations are reached than at weekends. Sunday, especially, is a day in which the concentrations of NO₂ are very low. We can also observe the existence of two peak times per day, with Thursdays and Fridays being the days when these peaks are highest. Although only one example week is shown in the figure, this pattern is repeated every week of the year, although there are differences between the different seasons of the year. We also observe a difference between the sensors located in urban areas and rural areas. The closer a sensor is located to the city, the higher the NO₂ concentration, especially during peak hours.

Another interesting pattern observed in our data is that public holidays have a similar influence as Sundays. The concentration of NO₂ is very low compared to the other days of the week. We also observe that those, known as “bridge days”, that is, those days trapped between holidays and weekends, have NO₂ concentrations very similar to Saturdays, that is, lower than the other days of the week, but not as low as Sunday. In Figure 8, we have plotted the NO₂ concentrations for three of our sensors (Unfortunately for sensor DETH081, it was not possible to retrieve the data for that date) during the week of October 1 to 5, 2018. As seen in Figure 8, despite being Wednesday (a day of very high concentrations), the concentrations are extremely low since October 3rd is a public holiday in Germany (German Unity Day). In Section 6.1, we will address more on this topic.

4. On the Use of Embedding Layer in Neural Network: Encoding Traffic

Although neural networks can be considered a fairly old concept in the field of artificial intelligence [28,29], for some years their popularity declined and become practically ignored. However, this began to change in 2006 when Dr. Geoffrey E. Hinton introduced the deep belief networks [30]. It completely revolutionised the area of neural networks, giving rise to deep learning. One of DL’s greatest contributions has been in the area of natural language processing [31], where the introduction of embeddings achieved unprecedented improvement. An embedding is a mapping of a categorical variable to a vector of continuous numbers. In the context of neural networks, embeddings are low-dimensional, learned continuous vector representations of discrete variables [32].

4.1. DNN + Embedding

In this study, we used a dense neuronal network with an embedding layer to encode calendar information. Dense Neuronal Networks (DNN) are fully connected networks. This means each neuron in a layer receives input from all the neurons in the previous layer, as outlined in Figure 9, which shows a representation of our network. Embeddings, as well as meteorological variables, serve as input features (features layer). Finally, we used two hidden layers with Relu-activation (Rectified Linear Unit). The output was one neuron with linear activation.

Embedding Layer for Calendar Features

The embedding layer was used to encode calendar information. We considered the following embedding variables:

• Hour: The categorical variable $hour$ takes values in $\left\{0\dots 23\right\}$, so in a one-hot-encoding it has dimension 24. For its embedding dimension, we chose six, which follows recommendations to use $25\%$ of the input dimension for the embedding space [33].
• Weekday: The categorical variable $weekday$ takes values in $\left\{0\dots N\right\}$, where N depends on the representation of holidays outlined below. Its dimension is two. In our study, we defined: weekday as $\left\{0\dots 9\right\}$, considering seven weekdays and three types of holiday; partial holiday, public holiday and bridge day (bridge, partial, and public holiday describe days with influence through public holidays; public is the actual public holiday, partial is a public holiday in only parts of Germany and bridge describes days between a public holiday and weekends). A list of holidays used in this study can be found in the Appendix A.
• Month: The categorical variable $month$ takes values in $\left\{1\dots 12\right\}$, its dimension is three.

The dimension of the vector embeddings was fixed in accordance with the recommendations by the authors of [33] for the use of embeddings in calendar variables.

5. Experimental Study

In this section, we describe the experimental study we carried out for the prediction of the NO₂ concentration in the city of Erfurt. Firstly, we will refer to the data that we used as well as the source from which they were obtained; next we will show the setup of our experiments, then the results and finally, our conclusions about the results.

5.1. Data

As indicated in Figure 9, two sets of data were needed: the historical NO₂ measurements and meteorological variables. The calendar model uses only the historical NO₂ time series and the meteorological model uses the historical NO₂ and the meteorological variables, a multivariate time series with 7 variables.

The basic calendar information (day of the week) was directly provided by the used programming language (Python). Additional information such as about partial holidays, public holidays and bridge days was provided as a dedicated data set (CSV file) to the model. The classification of the days is described in Appendix A.

For the time series data, an integrative approach was chosen. In the first step, external data were imported from external sources and then transformed into a unified data model. Afterwards, these data were used to train the model, using a common interface for data access. It turned out that the OGC SensorThings API [34] provides an intuitive data model and an easy-to-use interface to access time-series data. In this case study, we relied on historical NO₂ values from the European Environment Agency. This dataset was already available through the SensorThings API (https://datacoveeu.github.io/API4INSPIRE/datanests/ad-hoc.html and https://airquality-frost.docker01.ilt-dmz.iosb.fraunhofer.de/v1.1, accessed on 10 December 2022).

In addition, commercial weather data were used to provide the meteorological variables. Those data sources were imported into FROST^® (https://github.com/FraunhoferIOSB/FROST-Server, accessed on 10 December 2022), an open-source implementation of the SensorThings API. Our forecast model uses seven weather variables, which have been recommended by Bosch (Product Area Air Quality Solutions Passenger Car (PS/PAQ-PC), Robert Bosch GmbH) experts. The considered variables are the following:

• Wind speed (km/h);
• Wind direction (degree);
• Precipitation (mm/h);
• Temperature (°C);
• Pressure (hPa);
• Cape (J/kg);
• Radiation (W/m${}^2$).

Since all variables have a different scale, we used the Z-score normalisation, i.e., we computed $(x-\mu )/\sigma$ for each value x, where $\mu$ is the mean and $\sigma$ the standard deviation of the corresponding data.

Table 2. Training and testing period per sensor.

Sensor	Data Points	Training	Testing
DETH043	16959	2018, 2019, 2020 until August	September, October and November 2020
DETH020	25127	2018, 2019, 2020 until August	September, October and November 2020
DETH117	15967	2019, 2020 until August	September, October and November 2020
DETH081	15010	2019, 2020 until August	September, October and November 2020

shows the time periods for each of the considered sensors, where historical data are available. As can be seen, the last three months were used as a test sample and the rest of the data (all available data) were used to train the model. The chosen approach allows decoupling of the training of the model from data provisioning. There is an abstraction for the data source-specific interfaces: zipped CSV files from the European Environment Agency’s proprietary JSON-based interface for the meteorological variables. Only a standardized SensorThings API endpoint needs to be queried when training the model. In the future, it will be possible to exchange data sources (e.g., choosing a different provider for the meteorological variables) to obtain even better prediction results. In this case, only the import needed to be adapted, whereas the model itself kept unchanged.

5.2. Parameters, Data Structure and Models Configuration

The parameters and configuration of most of the models used in the study were obtained from those recommended by the authors of the papers studied. Detailed parameters are shown in

Table 3. Parameter and model configuration for methods used in our study.

Model	Parameters	Input Data
DNN	Table 4	model 1: calendar
		model 2: cal+met
LSTM	hidden layer = 2	cal+met
	neurons/layer = 64
	epochs = 30
	dropout = 0.4
	optimizer = Adam
	loss = MSE
LGBM_Bosh parameter	max_depth: −1	cal+met
	learning_rate: 0.005
	num_iterations: 4837
	feature_fraction:0.6
	bagging_fraction: 0.9
	bagging_freq: 5
LGBM_bayesian_opt	max_depth: 15	cal+met
	min_split_gain: 0.1
	num_iterations: 100
	feature_fraction:1.0
	bagging_fraction:1.0
	num_leaves:5
LGBM_Qadeer	max_depth:-1	cal+met
	learning_rate:0.005
	num_iterations: 4837
	feature_fraction:0.6
	bagging_fraction:0.9
	bagging_freq:5
Adaboost	base_estimator = Decision_tree	cal+met
	n_estimator = 50
	learning_rate = 1.0
LSTM-encoder-decoder	dropout = 0.4	cal+met
	loss = MSE
	epochs =30
	optimizer = Adam
	encoder_LSTM
	embedding layer = 1
	embedding layer neurons = 16
	encoder_rnn_hidden = 224
	decoder_LSTM
	embedding layer = 1
	embedding layer neurons = 16
	decoder_rnn_hidden = 224

cal+met: calendar variables and meteorological variables forecasting.

. In some cases where they were not mentioned, we used the default values, and in the case of DNN+embedding, we used those parameters recommended in [33].

Table 4. Dense neural network parameters.

Parameters	DNN
Model	Sequential
Hidden layers	2
Neurons per layer	60/60/1
Loss function	mse
Type of layer	dense
Activation output	linear
Activation hidden layers	relu/relu
Epoch	100
Optimizer	RMSprop(0.001)

shows the configuration of the DNN+embedding used in our study. It is important to mention that for the selection of the parameters of our DNN we did not carry out any process of selection/optimisation of the hyperparameters. The selection was based on a recent study and the authors considered that, for future work, an optimisation study of the hyperparameters and the architecture of the DNN should be carried out [33].

6. Results

As we saw in the previous section, most state-of-the-art methods only predict pollutant concentrations for the next hour, which is not sufficient for our use case. Our goal was to predict 24 h of NO₂ concentrations and thus make decisions that help avoid high NO₂ concentrations. That is why in our experimental study we defined three time horizons:

• Forecast the next 24 h;
• Forecast the next 72 h;
• Forecast the next 120 h.

As Table 2 shows, our test set is, in all cases, from September 2020 until the last day available in the data (in some cases November, in others 17 December). The training procedure followed was: starting on 1 September, we iteratively forecasted the next time horizon hours, we calculated the evaluation metric, mean absolute error (MAE Equation (1)), added the current testing period to the training set, and again we forecasted the next time horizon period. The idea behind this daily recalibration schema was to have a mean error of several testing periods (every time horizon).

$$MAE=\frac{1}{H}\stackrel{H}{\sum _{i=1}}\left|N_r-N_p\right|,$$

(1)

where H is the number of hours, $N_r$ is the real NO₂ concentration and $N_p$ is the predicted NO₂ value.

We used the standard scaler for the meteorological variables.

An important observation is that the results do not vary as the time horizon varies. This is because we used the real measurements of the meteorological variables and not the forecasts, the history of the meteorological forecast is not available in the Meteomatics API. In the final application, the model will be trained with the historical data (NO₂ and meteorological variables) but the forecast of NO₂ will be made from the forecast of the meteorological variables.

Table 5. Mean absolute error average for all the compared methods.

Sensor	Method	Data/Encode	24 h	72 h	120 h
DE_DETH043	DNN+embedding	model 1	9.1929	9.1248	9.3957
		model 2	7.1700	7.3858	7.4489
	LGBM	ordinal	7.2325	7.3216	7.4517
		one_hot	7.2789	7.4297	7.4600
	LGBM-BOSCH	ordinal	6.9420	7.0837	7.1495
	LSTM	input = 10	8.7743	8.4007	8.1686
		input = 24	10.7109	11.2327	10.8840
	Adaboost	onehot	13.2796	13.3503	13.1704
		ordinal	12.5644	12.3406	12.5564
	LSTM-AE	seq len 5	11.0249	14.0768	14.3268
DE_DETH020	DNN+embedding	model 1	6.5689	6.7591	6.8499
		model 2	4.9559	5.1095	4.9550
	LGBM	ordinal	5.2947	5.2884	5.3362
		one_hot	5.3856	5.4101	5.3858
	LGBM-BOSCH	ordinal	5.1039	5.1641	5.2147
	LSTM	input = 10	5.3348	5.3718	5.4127
		input = 24	6.4097	6.5072	6.2651
	Adaboost	onehot	8.2344	8.0955	7.9516
		ordinal	9.3091	9.1417	9.3261
	LSTM-AE	seq len 5	7.9955	9.3003	9.4265
DE_DETH117	DNN+embedding	model 1	6.6708	6.6076	6.7113
		model 2	4.3912	4.5071	4.4123
	LGBM	ordinal	4.3028	4.3722	4.4029
		one_hot	4.3391	4.4280	4.4166
	LGBM-BOSCH	ordinal	4.1809	4.2906	4.2971
	LSTM	input = 10	4.6209	4.7725	4.8600
		input = 24	5.9599	5.8841	6.0775
	Adaboost	onehot	7.4650	7.4633	7.5334
		ordinal	7.4651	7.3091	7.2455
	LSTM-AE	seq len 5	7.4113	7.7973	7.7239
DE_DETH081	DNN+embedding	model 1	7.5889	8.0022	8.1991
		model 2	6.4381	6.3969	6.8792
	LGBM	ordinal	7.4838	7.7298	7.9176
		one_hot	7.3926	7.5320	7.7081
	LGBM-BOSCH	ordinal	7.2872	7.5083	7.5210
	LSTM	input = 10	8.2826	7.9086	7.7629
		input = 24	11.3915	11.5506	11.3960
	Adaboost	onehot	13.3968	13.5412	13.4224
		ordinal	13.5186	13.6836	13.1394
	LSTM-AE	seq len 5	10.8096	15.0223	14.6538

shows the results of training the model with historical pollution data and meteorological variables for all the studied models. As can be seen in Table 5, the most competitive methods are model 2 of DNN+embeddings and the LGBM model used by Bosch, which significantly outperform the rest of the models.

Despite the Bosch LGBM model and model 2 using DNN+embeddings behaving very similarly, for the DETH081 sensor the difference is significant when the DNN+embedding outperforms the LGBM model by almost 1 point. In this sense, we must point out that this was what we were waiting for. As we described in Section 3 and as observed in Figure 2, sensor DETH081 is very close to a road with a lot of traffic, which makes it the sensor (of the four studied) most affected by traffic.

These results confirm our theory that, for those sensors that cover areas highly affected by traffic, the use of embedding variables to encode the calendar information in a dense neural network is a significantly superior solution to the rest of the methods with which we have compared it in this study.

In order to offer more details about the performance of our models, below we will present some plots, where it is possible to see how our model is able to predict (quite accurately) the month of September 2019, a month in which quite unusual behaviour of NO₂ concentrations was observed.

In Figure 10, Figure 11, Figure 12 and Figure 13, we show the real and predicted values for the month of September 2019. The time horizon used in the prediction was 120 h (5 days). As can be seen, despite the fact that our model has a fairly accurate behaviour in the prediction, there are some peaks in which our prediction is very far from the real value. Looking in detail at those peaks, and comparing them with similar days, we discover that these values are exceedingly high and outliers. Something out of the ordinary (not meteorological) could cause this increase in the NO₂ level. From our research, we know that traffic and meteorologic conditions are the main causes of variations in the levels of NO₂, leading us to the assumption that an increase in traffic may be the cause of these high levels of NO₂.

However, we have drawn an important conclusion from this unusual behaviour: for future improvements of our model, it would be very beneficial to have a new input variable, related to local festivities (carnivals, concerts, festivals), as well as accidents that occurred in the neighbourhood of the sensor, for the model. Although it is obvious that accidents are not predictable, and therefore this variable could not be used as input to the model to make future predictions, it could be used to explain unusual peaks.

Despite the four studied sensors registering the high outliers, sensors DETH043 and DETH081 reached the highest values, with almost 120 $\mathsf{\mu }$g m${}^3$, and sensor DETH117 (the one located in the rural area) registering the lowest peaks.

6.1. Interpretability through the Embedding Space

The embedding vectors resulting from neural network training are often very useful for finding behaviour patterns in categorical variables that cannot be distinguished with the naked eye. Following this assumption, we show the use of the embedding vector obtained during the training of the DNN to graphically understand how the models use the calendar information in the forecast. To visualise the resulting embedding vectors, we coded some functions that recreated what Tensorflow Projector does https://projector.tensorflow.org/, accessed on 10 December 2022).

Figure 14 shows in three dimensions the resulting embedding vectors for the 24 h. As can be seen, the hours form a cycle, but they do not behave like a clock.

In Figure 15 we show the projection of the resulting embeddings vectors for the dimension “day of the week”. In this case, the way in which the embedding vectors have been grouped is very interesting. Sundays and holidays are very close as well as Saturdays and bridge days; the weekdays have also created two well-defined groups, which makes total sense and is in total correspondence with what was observed in the study carried out in Section 3.

Finally, Figure 16 shows the projection of the embeddings obtained for the months of the year. In this case, although not all the months belong to well-defined groups, it can be seen, for example, that the winter months are quite close together, although partially mixed with the autumn months. Something similar happens with the spring and summer months.

As has been observed in the projection of the embeddings resulting from the training of our dense neural network, the use of embeddings not only guarantees a better encoding of the categorical variables within the neural networks but also gives us some clues about the interpretability of the results. The grouping of the categories observed in Figure 14, Figure 15 and Figure 16 can help users of our model to understand, for example, a very low forecast for a Wednesday (day of very high NO₂ levels) only if that Wednesday is a holiday. It can also be useful in making decisions related to reducing NO₂ levels.

6.2. Our Contribution to the Sustainable Development Goals

The Sustainable Development Goals (SDGs), also known as the Global Goals, were adopted by the United Nations (UN) in 2015 as a universal call to action to end poverty, protect the planet, and ensure that by 2030 all people enjoy peace and prosperity [35].

The 17 SDGs enunciated by the UN have become a priority for many countries, and in this sense Germany is a leader within them. Of the 17 goals stated by the UN, we believe that our work contributes to three of them (highlighted below):

• No poverty;
• Zero hunger;
• Goodhealthandwell-being;
• Quality education;
• Gender equality;
• Clean water and sanitation;
• Affordable and clean energy;
• Decent work and economic growth;
• Industry, innovation and infrastructure;
• Reduced inequalities;
• Sustainablecitiesandcommunities;
• Responsibleconsumptionandproduction;
• Climateaction;
• Life below water;
• Life and land;
• Peace, justice and strong institution;
• Partnerships for the goals.

Goodhealthandwell-being: The reasonable, controlled and environmentally committed use of transportation, as well as the measures taken by local governments after knowing the forecast of pollutants in the air, will undoubtedly result in big health benefits.

Sustainablecitiesandcommunities: Using an application such as the one suggested in this research can help considerably with achieving more sustainable cities. Letting the population and local governments know the quality of the air they breathe, and at the same time making them participate and be responsible for reducing polluting gases, will achieve more sustainable cities and citizens more committed to the environment.

Climateaction: From our point of view, this application puts more responsibility in the hands of citizens and local governments, telling them in a closer way, “you can do a lot for the planet”. Somehow it is a popular thought to believe that climate change is a matter for big industries and governments at very high levels and that one person cannot do anything. However, this application goes to show that if "you only use your bicycle” or “take public transport” (that the government has reduced the price of), you will be able to breathe cleaner air; then, indeed, we are all taking action against climate change and taking care of the planet.

7. Conclusions and Future Work

In this paper, we have presented a study to forecast NO₂ concentrations in the city of Erfurt, Germany, 1, 3 and 5 days in advance. We used some of the most significant methods within the state-of-the-art to forecast pollutants. We also introduce the use of DNN using embedding variables to encode calendar information. The comparative study carried out shows the most competitive methods the LGBM model with the hyperparameters currently used by the “Robert Bosch GmbH” and our proposal.

Although for three of the four sensors studied, there is no difference regarding the Bosch LGBM model and ours, for one of the sensors (the one that is very close to a highway with a lot of traffic) this difference is significantly in favour of our model. This result corroborates our hypothesis and allows us to conclude that, in the forecast of the NO₂ concentration time series, the use of embedding variables to encode the calendar information, in a dense neural network, also encodes the traffic behaviour in a very efficient way. For this reason, we highly recommend our proposal for all those pollutants forecast applications (associated with traffic), in which the traffic data are unknown, as in our case. The second important conclusion that we can draw from our study is that the visualisation of the embedding vectors resulting from the training of the DNN can be considered a very useful tool for finding relationships between categorical variables and associated concepts, and for making decisions that in some way contribute to the reduction of emissions. In future work, we intend to improve our model by using the calendar service currently being developed by the Fraunhofer-IOSB, incorporating new meteorological variables, and optimising the DNN hyperparameters as well as the network architecture.

We like to believe that, in a very modest way, the use of our models by users at different levels (from high government officials with decision-making power to citizens who can decide whether to use a bicycle or a car) will contribute to reducing emissions of NO₂.