Mutual Information Input Selector and Probabilistic Machine Learning Utilisation for Air Pollution Proxies

Abstract

An air pollutant proxy is a mathematical model that estimates an unobserved air pollutant using other measured variables. The proxy is advantageous to fill missing data in a research campaign or to substitute a real measurement for minimising the cost as well as the operators involved (i.e., virtual sensor). In this paper, we present a generic concept of pollutant proxy development based on an optimised data-driven approach. We propose a mutual information concept to determine the interdependence of different variables and thus select the most correlated inputs. The most relevant variables are selected to be the best proxy inputs, where several metrics and data loss are also involved for guidance. The input selection method determines the used data for training pollutant proxies based on a probabilistic machine learning method. In particular, we use a Bayesian neural network that naturally prevents overfitting and provides confidence intervals around its output prediction. In this way, the prediction uncertainty could be assessed and evaluated. In order to demonstrate the effectiveness of our approach, we test it on an extensive air pollution database to estimate ozone concentration.

1. Introduction

Air pollution describes the presence of harmful substances in the atmosphere, which are detrimental to human health as well as the Earth’s climate. The World Health Organization (WHO) estimates that ~7 million people die every year from exposure to fine particles in polluted air. Such particles induce a variety of diseases such as strokes, heart diseases, lung cancer, chronic obstructive pulmonary diseases and respiratory infections, including pneumonia [1]. Furthermore, air pollution trace gases, such as ozone, has proven to accelerates the progression of emphysema of the lung [2]. In urban environments, air pollution is mostly generated from vehicle emission as well as industrial and domestic fossil fuel combustion [3]. Such processes, i.e., the extraction and burning of fossil fuels, also emit carbon dioxide (${\mathrm{CO}}_2$), which is found to be a key driver of climate change [4,5].

Air quality indicators are typically monitored by environmental or government authorities using networks of fixed monitoring stations [6], equipped with instruments specialised for measuring a number of pollutants, such as carbon monoxide ($\mathrm{CO}$), nitrogen oxides (NO_x), sulphur dioxide (${\mathrm{SO}}_2$), ozone (${\mathrm{O}}_3$) and particulate matter (PM) [7]. In practice, a comprehensive measurement covering a large area may not always be possible as air pollutant analysers are generally complicated, bulky and labour-intensive [7,8]. Furthermore, instrument failure, fault in data acquisition or data corruption often result in missing data during research campaigns or continuous measurements [9,10,11]. If the data gap is relatively large, interpolation methods become ineffective. As a result, these problems pose a significant obstacle for comprehensive air pollution data analysis and time series prediction scheme.

There are several solutions in order to deal with the aforementioned challenges, including low-cost air quality sensor networks [12], chemical transport simulation models [13] and satellite remote sensing data [14]. Here, we propose the solution by developing a pollutant proxy that is also known as a model or an estimator. A pollutant proxy can be defined as a mathematical model that estimates an unobserved air pollutant using other measured variables. Such a proxy acts as a virtual sensor or an expert system to fill missing data or to substitute one or more real measurements, thus the number of operated instruments can be minimised leading to a reduction in operational cost as well as the operators involved. Therefore, the area with pollution information can also be widened. A pollutant proxy is also useful to support scientists in studying the impact of relevant variables to pollutants. Their relationship can be simulated and then analysed.

Physics-based approaches are usually first to solve the aforementioned problems. These are models based on the fundamental description of atmospheric physical and chemical processes, such as Lagrangian and Eulerian models [15]. However, this approach is very computationally demanding. Simple statistical predictive models are usually the next attempt to solve the aforementioned problems, such as described in Clifford et al. [16], Mølgaard et al. [17], von Bismarck-Osten et al. [18], Khalil et al. [19], Alghamdi et al. [20]. However, these proxies may not always be suitable or perform well due to the poor input choice, nonlinearity in data and the complexity in establishing appropriate physics-based models.

As an alternative approach, data-driven methods have recently gained much attention for developing pollutant proxies. For examples, various types of data-driven ${\mathrm{O}}_3$ proxies were developed based on adaptive neuro-fuzzy inference by Taylan [21], artificial neural network (ANN) by Gao et al. [22], Arroyo et al. [23], support vector machine (SVM) by Ortiz-García et al. [24] and based on ANN and SVM by Luna et al. [25]. However, those methods mainly attempt to use all input combinations to obtain the best model performance during the training phase. This method may not always be effective for large data sets because of the computational cost and analysis issues. Recently, a large number of instruments involved in research campaigns as well as continuous measurements produces massive amounts of data. Additionally, many developed methods are mostly based on deterministic data-driven approaches where these proxies only estimate mean values of pollutant concentrations. Consequently, these proxies are unable to evaluate and assess the estimation uncertainty. For instance, if there is a sudden increase of pollutant concentration estimation, it is not possible to assess its confidence interval [26,27]. Therefore, additional measurements and validation may be required to confirm the estimation.

The main contributions of this paper lie in dealing with the aforementioned problems as part of design solutions. First, we apply mutual information (MI) method for selecting appropriate data as proxy inputs. This method is also complemented by the evaluation of performance metrics as well as data loss accumulation to select the number of maximum inputs involved. Second, a probabilistic machine learning (ML) method is implemented, where in this case we use a Bayesian neural network (BNN). Neural networks (NNs) have been a popular choice among ML methods for approximating complex functions [28] and have been used in a wide variety of problems in various disciplines [29]. However, standard NNs general formulation typically gives single point based output estimates. The Bayesian inference implementation in the NNs does not solely prevent overfitting in the training phase, but it also provides confidence interval around its prediction [30,31,32]. Third, this work is a continuation of our previous work in developing ${\mathrm{O}}_3$ proxy [20] that was based on physics-based approaches using an extensive air pollution measurement campaign database. In this work, using the same data sets, we demonstrate that the proposed data-driven-based proxy outperforms the previous developed proxy. Finally, we discuss the practical implementation on deploying the developed proxy including the compromise between an automatic input selector based MI and the number of used instruments for minimising the costs as well as the number of operators involved, so as to gain maximum benefit in practice.

2. Methods

This section describes our proposed methodology for developing a pollutant proxy. Figure 1 presents a generic data-driven-based methodology for a pollutant proxy development and deployment. The concept is mainly based on two key approaches. The first is called a mutual information (MI) approach that is used to select the most relevant proxy inputs. The second is a probabilistic machine learning (ML) method that is used to model an air pollutant.

Having a large amount of field measurement data, the first step is to explore the available database. For example, data analysis is typically performed to understand data statistical properties, such as mean, standard deviation, minimum and maximum values of each variable, the number of missing data, the linear correlation, etc. From data exploration, it is known how one measured variable is related to other measured variables. Standard correlation methods, such as Pearson correlation coefficient (PCC), is typically applied. However, as air pollution is a complex process, the relationship between the measured variables may not always be linear. Therefore, we complement the correlation analysis by an information theory-based technique, named mutual information (MI), to search relevant measured variables correlated linearly and nonlinearly to an air pollutant to be modelled.

From the data exploration, it is known that there is a substantial amount of missing data. To solve the problem in practice, an additional guidance method is required for inputs selection. When generating a proxy, we are often limited to the measurement period during which data of all variables exist (i.e., the data with same time index). This leads to fewer data points available as the inputs for training phase of the pollutant proxy. The missing data scenario results in significant data loss for the training purposes. Furthermore, the use of a large number of inputs typically increases the model complexity leading to poor proxy performance. Limiting the number of inputs also allows the proxy to be used flexibly without relying on many other measurements in practice. Therefore, it is vital to consider these effects when determining the number of inputs involved in modelling, shown as the box of performance and data loss monitoring in Figure 1. The selected input data is then used for ML training and validation data. Finally the results are validated and analysed. This whole process can be called pollutant proxy development. The development results in the pollutant proxy where it can then be deployed for filling the missing data in the database and/or act as a virtual sensor.

2.1. Input Selector: Mutual Information

The mutual information (MI) of two random variables is a measure of the mutual dependence between these two variables. MI is thus a method for measuring the degree of relationship between data sets. This method has been widely used in various modern disciplines of science and technology, such as medical imaging [33], search engine [34] and DNA sequencing [35]. In the field related to atmospheric and environmental sciences, MI has been introduced to large data sets to detect nonlinear relationship between new-particle formation and other ambient variables in Hyytiälä Forest, Finland, as described in our previous work [36]. In this case, we adopt the same strategy for finding the relationship between a pollutant proxy and other measured variables.

The concept of MI is depicted in Figure 2. Suppose X and Y are the measured variables and the variable of interest. The area covered by red and orange is the entropy of X, $H(X)$, whereas the area covered by yellow and orange represents the entropy of Y, $H(Y)$. The red area is the conditional entropy of X given by Y, $H(X|Y)$, whereas the yellow area is the conditional entropy of Y given by X, $H(Y|X)$. The area contained by both circles is the joint entropy $H(X,Y)$ and the orange area is the mutual information between X and Y, $I(X;Y)$. Therefore, MI can be found by calculating

$$\begin{array}{cc}\hfill I(X;Y)& =H(X,Y)-H(X|Y)-H(Y|X)\hfill \end{array}$$

(1)

$$\begin{array}{cc}\hfill & =H(X)+H(Y)-H(X,Y)\hfill \end{array}$$

(2)

As used in Zaidan et al. [36], we also adopt a nearest-neighbour MI implementation based on Kraskov et al. [37]. The nearest neighbour concept assumes the space $Z=(X,Y)$. We can rank for each point $z_i=(x_i,y_i)$, its neighbours by distance $d_{i,j}=\Vert z_i-z_j\Vert$ and similar rankings can also be done for the subspaces X and Y. So, the distance can then be found by finding the maximum norm: $\Vert z-z^{\prime }\Vert =max\{\Vert x-x^{\prime }\Vert ,\Vert y-y^{\prime }\Vert \}$ where the distance from $z_i$ to its kth neighbour is denoted by $ϵ(i)/2$. The MI estimation is then given by

$$I(X;Y)=\psi (k)-\langle \psi (n_x+1)+\psi (n_y+1)\rangle +\psi (N)$$

(3)

where the symbol $\psi (.)$ represents the digamma function. N is the number of all data points and k constitutes the number of nearest neighbours (the user choice). Parameters $n_x$ is the number of point $x_j$ whose distance from $x_i$ is strictly less than $ϵ(i)/2$, and similarly for y instead of x. The symbol $\langle .\rangle$ denotes the averages both over all data points N. The $I(X;Y)$ can then be normalised between 0 and 1 by a scaling factor [38], so the final MI formulation is given by

$$\widehat{I}(X;Y)=\mathrm{sign}\left[I(X;Y)\right]\sqrt{1-exp(-2|I(X;Y)|)}$$

(4)

where symbol sign represents a signum function and notation $|.|$ is the absolute value. The mathematical details of the used MI method can be found in Kraskov et al. [37], and its implementation in this field can be found in Zaidan et al. [36].

2.2. Probabilistic Machine Learning: Bayesian Neural Networks

Machine learning (ML) infers plausible models to explain observed data where the models are capable to make predictions about unseen data and take decisions that are rational given these predictions. However, there are many forms of uncertainty in modelling, such as measurement noise and model general structure. Probabilistic modelling framework takes into account the uncertainty in modelling by using probability theory to express all forms of uncertainty [39]. One of the most popular ML methods is neural networks [28]. The rise of deep learning and related methods have made neural network based approaches become even more popular [40,41]. However, their standard implementations are mostly based on deterministic methods, providing point-based estimations [42]. Although, there are many strategies in preventing overfitting, such as cross-validation, early stopping and regularisation, the standard implementation does not provide confidence-interval around its prediction. A Bayesian neural network (BNN) is a type of probabilistic modelling method [43], and here we adopt this method for estimating pollutant concentration. However, other types of probabilistic models may also be applicable if they are implemented appropriately.

BNNs were popularised by MacKay [44], Neal [45]. The methods have been advancing since then to cope with many issues, such as challenging inference and computational costs. Several improved BNN versions have been developed, including variational inference [46], sampling-based variational inference [47] and expectation propagation [48]. In this case, we adopt one of the most recent version of BNNs where it uses dropout applied before every weight layer as Bayesian approximation [49]. For the case of a single hidden layer, dropout NN structure can be shown mathematically as

$$\widehat{\mathbf{y}}=\sigma (\mathbf{x}({\mathbf{z}}_1{\mathbf{W}}_1)+\mathbf{b})({\mathbf{z}}_2{\mathbf{W}}_2)$$

(5)

where $\widehat{\mathbf{y}}$ and $\mathbf{x}$ are output and input models, respectively. The notation of ${\mathbf{W}}_1$ is a $Q\times K$ weight matrix connecting the first layer to the hidden layer and ${\mathbf{W}}_2$ is a $K\times D$ matrix connecting the hidden layer to the output layer. The symbol $\mathbf{b}$ is K dimensional vector biases, whereas $\sigma (.)$ is a nonlinear activation function. Dropout can be applied through the sampling of two binary vectors ${\mathbf{z}}_1$ and ${\mathbf{z}}_2$ of dimensions Q and K, respectively. They are modelled as Bernoulli distribution with some parameters $p_i\in [0,1]$. The process can be repeated for multiple layers. To optimise the weights, $L_2$ regularisation weighted by some decay $\lambda$ is used, given by

$${\mathcal{L}}_{\mathrm{dropout}}=E+\lambda \sum _{i=1}^L\left(\parallel {\mathbf{W}}_i{\parallel }_2^2+{\parallel {\mathbf{b}}_i\parallel }_2^2\right)$$

(6)

where L is the total number of layers and E is Euclidean loss, defined by

$$E=\frac{1}{2N}\sum _{n=1}^N\parallel {\mathbf{y}}_n-{\widehat{\mathbf{y}}}_{\mathbf{n}}\parallel$$

(7)

that is, simply the loss function between the predicted and the real data. New realisations are sampled for the binary vectors $z_i$ for every input point and every forward pass when evaluating the model’s output, and the same values are also used in the backward pass. The architecture of dropout NN based method is depicted in Figure 3. The use of dropout, applied before every weight layer, trained in deep NN with arbitrary depth and nonlinearity has been shown as approximate Bayesian inference in deep Gaussian process. As the posterior distributions of weight matrices, $\mathbf{W}$, are intractable, variational inference [50,51] is applied to approximate the distributions. The details of the approximate predictive distribution and other mathematical formulations are described in Gal and Ghahramani [49], whereas the implementation, such as designing network architecture and its inference, is done in an open source machine learning library called PyTorch [52].

3. Case Study

In this section, we explain a case study, including the sampling site and the data sets used for evaluating the proposed method. The proposed pollutant proxy is tested using an extensive measurement data obtained in Jeddah, Saudi Arabia. In particular, we choose to develop a proxy of Ozone (${\mathrm{O}}_3$) concentration, but the proposed method is also generic for other types of air pollutant.

The used data set is one of the most comprehensive measurement in the Middle Eastern region [54]. The sampling was performed in an urban background area of Jeddah city, Kingdom of Saudi Arabia (Figure 4). The site is located at 21.4869° N, 39.2517° E and the altitude is 38.7 m above sea level. The measurement site is about 105 m and 1.7 km to the nearest road and the major highway, respectively. The measurements were performed at the height of 3.5 m and 6.7 m for gaseous air pollutants and meteorological parameters above the ground level, respectively. The sampling originally took place from 31 July 2011 to 28 February 2013. However, the ${\mathrm{O}}_3$ concentration data is only available for 258 days.

A variety of scientific instruments were involved in the measurement campaign. ${\mathrm{O}}_3$ concentration was monitored by an UV Absorption Ozone Analyser (Model 400E, Teledyne Technologies Company, San Diego, CA, USA), whereas $\mathrm{NO}$, ${\mathrm{NO}}_2$ and ${\mathrm{NO}}_{\mathrm{x}}$ concentrations were measured using a chemiluminescence $\mathrm{NO}$/${\mathrm{NO}}_2$/${\mathrm{NO}}_{\mathrm{x}}$ analyser (Model 200E, Teledyne Technologies Company, San Diego). Syntech Spectras BTEX analyser GC 955, type 600 (Syntech Spectras, Groningen, the Netherlands) was used to measure a specific volatile organic compound (VOC) concentrations, that are benzene, toluene, ethylbenzene and xylenes (BTEX). Particulate matter concentrations, such as PM${}_{10}$, PM${}_{2.5}$ and PM${}_1$, and the number concentration (TC) of particles in the diameter size range of $0.25$ to 32 $\mathsf{\mu }$m were measured with one-minute average by an optical scattering spectrometer (EDM-180D, Grimm Aerosol, Germany). A solar radiation sensor (Vantage Pro2™Accessories, Davis Instruments, Hayward, CA, USA) was used for continuous measurement of solar radiation. Finally, meteorological variables, such as wind speed, temperature and relative humidity, were measured using Lufft WS600-UMB Compact Weather Station. The detailed explanation of the sampling site, data sets and the instruments used in this research campaign can be found in the previous works by Alghamdi et al. [54,55], Hussein et al. [56].

${\mathrm{O}}_3$ concentration is the major gas component of air pollution in cities. Among other photochemical oxidants, ${\mathrm{O}}_3$ is one of the widely studied subjects worldwide under the category of air pollution [20]. For example, air pollution in Jeddah was investigated by analysing temporal variations of ${\mathrm{O}}_3$ and ${\mathrm{NO}}_{\mathrm{x}}$ [54] and ${\mathrm{O}}_3$ formation affected by benzene, toluene, ethylbenzene and xylenes [55].

Several reasons motivate this study to establish an ${\mathrm{O}}_3$ proxy as the case study. First, there are missing values spread across the data, including a large number of ${\mathrm{O}}_3$ concentration data, due to the instruments maintenance as well as other issues [56]. The proxy is expected to contribute in filling the missing data to allow better air pollution analysis. Second, the cost operation of official air pollution monitoring stations is typically high due to the number of instruments used. To substitute ${\mathrm{O}}_3$ monitoring, the proxy can be deployed to mimic the real measurement based on “Ozone Analyzer”. In this way, the number of instruments involved can be minimised. As a result, the operational cost can be reduced and the number of monitoring stations can be scaled up. Finally, human involvement is typically required to operate some instruments, including “Ozone Analyzer” [57]. The ${\mathrm{O}}_3$ proxy acts as a virtual sensor where the number of operators can be reduced. The aforementioned proxy advantages can be even greater if the proxy is deployed in one or several long-term continuous measurement stations.

4. Results and Discussion

This section comprises three subsections. First, we describe data analysis to get insight about the available database. Second, the results of proxy implementation and its performance are presented and explained. Finally, we discuss about the proxy deployment in practice.

4.1. Data Analysis

Linear correlation analysis is performed on every measured variables in the database. As described in Section 2, we also perform an additional MI analysis to ensure all possible relationships are captured between measured variables, either linear or nonlinear. Pearson correlation coefficient (PCC) is used here representing linear correlation method whereas mutual information (MI) acts as a nonlinear method. To make both methods comparable, as described in Zaidan et al. [36], we take absolute values of PCC because MI does not capture the correlation direction. In this case, the correlation direction is insignificant to be analysed because the air pollutant proxy will be modelled based on a black block approach (i.e., ML). Figure 5 shows the correlation analysis in the form of matrix plots. It can be seen that the linear correlation method (a) does not find strong correlation between ${\mathrm{O}}_3$ and several variables that are known to be connected to ${\mathrm{O}}_3$ formation, such as ${\mathrm{NO}}_2$ and ${\mathrm{NO}}_{\mathrm{x}}$. On the other hand, MI method (b) demonstrates that ${\mathrm{O}}_3$ presents good connection to these two variables.

From the MI scores, the variables that share the highest correlation to ${\mathrm{O}}_3$ are ${\mathrm{NO}}_{\mathrm{x}}$, wind speed (WS), m,p-Xylene (TNX) and ${\mathrm{NO}}_2$. It is well known from previous studies [20,58,59] that ${\mathrm{O}}_3$ is strongly connected to $\mathrm{NO}$ and ${\mathrm{NO}}_2$ as well as ${\mathrm{NO}}_{\mathrm{x}}$ concentrations. Furthermore, TNX was found as the largest contributor to ${\mathrm{O}}_3$ formation [55]. However, the relationship between these concentrations cannot be captured well by PCC analysis as shown in Figure 5.

Figure 6 presents the scatter plots of some selected variables. It can be seen that the three variables (e.g., $\mathrm{NO}$, ${\mathrm{NO}}_{\mathrm{x}}$ and TNX) are shaped to a specific pattern (i.e., exponential functions), indicating that they correlate nonlinearly to ${\mathrm{O}}_3$ concentration (these variables also behave nonlinearly in logarithmic scale). Their linear correlation coefficients and MI scores are also not similar. Nevertheless, both methods are able to capture good linear correlation between ${\mathrm{O}}_3$ and wind speed (WS) at R${}^2=0.76$ and MI = 0.79. WS has been known to affect ${\mathrm{O}}_3$ concentration in urban area as described in previous studies, such as explained by Hidy [60] and Klein et al. [61]. In summary, the use of MI allows the detection of linear and nonlinear correlation between variables of interest where the correlated variables can then be used as potential inputs for the pollutant proxy.

4.2. Performance Analysis

The results of the previous subsection justify our choice to utilise the strength of MI method for proxy inputs selection as proposed in Section 2. This subsection presents several results obtained from the proposed proxy development.

Figure 7 presents the rank of MI score of the measured variables correlated to ${\mathrm{O}}_3$ concentration. The variables with high MI score indicate the higher association with the ${\mathrm{O}}_3$ concentration. Based on the bar chart, we can select the appropriate inputs for training the pollutant proxy. We first explore the possibility to divide the proxies into two: day and night proxies as previously proposed in our previous work [20], and then also generate one single full day proxy. Therefore, MI between ${\mathrm{O}}_3$ concentration and other measured variables are applied on data sets for entire day, day only and night only. Figure 7 shows the MI level between ${\mathrm{O}}_3$ and other measured variables for two main types of proxy. The subplot (a) is the MI level for a full day analysis, whereas the subplots (b) and (c) constitute the MI level for day and night measurements, respectively. The analysis for the subplot (a) is used to select the inputs for the single ${\mathrm{O}}_3$ proxy, whereas information on the subplots (b) and (c) are utilised to select the divided ${\mathrm{O}}_3$ proxy. It is noted that the bar chart results also support the scientific findings discovered earlier in the previous section.

All proxies use at least two inputs at the first modelling attempt and then the inputs fed into BNN training following the order as shown in Figure 7. In order to evaluate the developed proxies, three types of performance metric are used. The first metric is called mean absolute error (MAE). It has a simple interpretation as the average absolute difference between the predicted proxy values ($\widehat{\mathbf{y}}$) and the real measurement data points ($\mathbf{y}$). It can be defined mathematically as

$$\mathrm{MAE}=\frac{{\sum }_{i=1}^n|{\widehat{y}}_i-y_i|}{n}$$

(8)

where $\widehat{y}$, y and n are the predicted value from proxy, the real measurement value and the number of data points, respectively. The second metric is root-mean-squared error (RMSE) that is also known as the standard deviation of the residuals (prediction errors). It is given by

$$\mathrm{RMSE}=\sqrt{\frac{{\sum }_{i=1}^n{({\widehat{y}}_i-y_i)}^2}{n}}$$

(9)

The third metric is called coefficient of determination, denoted by R${}^2$. It provides a measure of how well observed outcomes are replicated by the proxy, based on the proportion of total variation of outcomes explained by the proxy. It can be defined as

$${\mathrm{R}}^2=1-\frac{{\sum }_{i=1}^n{({\widehat{y}}_i-y_i)}^2}{{\sum }_{i=1}^n{(y_i-\overline{y})}^2}$$

(10)

where $\overline{y}$ is the mean of the number of measurement data points, $\mathbf{y}$.

Figure 8 shows proxy performance metrics and the percentage of data loss. The Figure 8a demonstrates the proxy performance after incorporating different number of inputs into BNN, including MAE, RMSE and R${}^2$. It is known that incorporating more inputs into ML models improves their performance generally, as shown in Figure 8a. Figure 8b demonstrates that after incorporating four input variables, the model performance still improves, but the number of data loss increases significantly, i.e., to be almost 80% because the need to find some common training data from all variables used in the training process. In a real-world problem, this is not always possible, some data points may be missing from several measurements at different time frames. As a result, the number of data points may be very less when we take many inputs into the ML training process. Using the remaining data that is less than 80% may not be optimal in data-driven approach because the use of more data is a key in proxy training. The good performance obtained may be because the number of validation data points becomes much fewer. It can also be seen that when the number of inputs is increased to be 10, the proxy performance deteriorates rapidly because there are not enough validation points at all.

Furthermore, the use of many input variables is also impractical when deploying the proxy. The deployed proxy will require many instruments working simultaneously at the same location to supply the same type of input data used in the training. Using many inputs from instruments indicates that the developed proxy is becoming less independent from real measurements. The real value of having a pollutant proxy will then be less because the deploying cost would rise. Therefore, as a compromise, we determine to select the number of inputs to be four. In this case, the number of missing data excluded can be minimised and we can still gain relatively good proxy performance.

After all, we decided to exclude the two separate proxies, i.e., day and night models. The main reason is that the proxy performances do not vary significantly between the single proxy and the coupled proxies as shown in Figure 8. Therefore, for simplification, we will present the results of single proxy only from now on.

A brute-force method is used to search the most optimal architecture of the BNN structure. In this case, we found that the best network consists of one hidden layer with rectifier activation functions and linear output layer. The hidden layer comprises 100 neurons, but the “drop-out” method (with 50% initial drop-out rate) is also applied as explained earlier to remove unnecessary neurons. For optimisation, an adaptive learning rate optimisation algorithm (Adam) [62] is used for tuning the BNN parameters. Once the top four inputs are selected, we applied cross-validation methods to find the optimal training/testing data that generate the best BNN. However, we found that this implementation does not result in significant difference in term of performance. It is also preferred to have continuous training and testing data in this case. Hence, we determine the used training data is from 21 December 2011 until 21 July 2012 whereas the validation data is selected from 22 July 2012 until 10 October 2012. This selection constitutes 70% training and 30% testing data. In this work, we do not compare the proposed probabilistic ML method with other ML methods, as the focus emphasises the concept of proxy development. The concept should be generic to be applied on other types of air pollutant proxy with different ML methods.

As discussed earlier, the objective of deploying proxy is to reduce the number of operated instruments in an air pollution station, and then to minimises the cost as well as the operators involved in the measurement processes.

Table 1. The above table shows that the combination of the best four variables to understand the impact of proxy performance by reducing the required instruments for computing ${\mathrm{O}}_3$ proxy.

Proxy Inputs	MAE	RMSE	R2
${\mathrm{NO}}_2$–${\mathrm{NO}}_{\mathrm{x}}$	7.154	86.621	0.703
${\mathrm{NO}}_2$–${\mathrm{NO}}_{\mathrm{x}}$–WS	5.594	50.409	0.83
${\mathrm{NO}}_2$–${\mathrm{NO}}_{\mathrm{x}}$–WS–TNX	5.192	48.564	0.84

displays the performance metrics of different inputs combination from the selected proxy inputs. Please note that the inputs of ${\mathrm{NO}}_2$ and ${\mathrm{NO}}_{\mathrm{x}}$ are not separated since they are measured from the same instrument. From Figure 5, it is found that TNX correlates highly with ${\mathrm{NO}}_{\mathrm{x}}$ which is another input of the proxy. Therefore, this variable can be discarded. It can be seen in the table’s last row that, by excluding TNX input, a very similar proxy performance can still be gained. This means that the optimal ${\mathrm{O}}_3$ proxy can be deployed by utilising only two instruments: the $\mathrm{NO}$/${\mathrm{NO}}_2$/${\mathrm{NO}}_{\mathrm{x}}$ analyser and a WS measurement device. The following results will then present only the three inputs based proxy that are $\mathrm{NO}$, ${\mathrm{NO}}_{\mathrm{x}}$ and WS.

Figure 9 presents a fraction of the time series results of real measurement of ${\mathrm{O}}_3$ concentration (red) and its proxy (blue) as well as its confidence interval (light green). Even though the prediction does not fit perfectly the real measurement data, some parts of predicted uncertainty capture the real data. Furthermore, it can be seen that the ${\mathrm{O}}_3$ proxy tracks very well the ${\mathrm{O}}_3$ concentration pattern, which is a good successful indicator from the developed proxy performance. The full time series data is not displayed because it is visually unclear to plot the whole time series of validation data set.

To understand the overall performance of developed model, we produce a regression plot and an error histogram as shown in Figure 10. Regression plot, subplot (a), displays the scatter plot between the measured ${\mathrm{O}}_3$ (x-axis) and the estimated ${\mathrm{O}}_3$ (y-axis). The light blue represents 2$\sigma$ uncertainty around the mean of ${\mathrm{O}}_3$ predictive distribution. It can be seen that the majority of ${\mathrm{O}}_3$ estimations lie within 2$\sigma$ of uncertainty. Furthermore, the obtained R${}^2$ performance is at $0.83$ on the scatter plot that is considerably higher than considering the simple dual model (day and night) as our previous work presented (R${}^2\approx 0.3$) by Alghamdi et al. [20]. The error histogram, subplot (b), displays the difference between the measured and estimated ${\mathrm{O}}_3$ concentration. The evaluation through error histogram is also promising because it follows a Normal distribution where the peak is centred around zero with standard deviation is only ~5 parts per billion (ppb).

4.3. Discussion

As mentioned earlier, in addition of filling missing data, an air pollutant proxy is advantageous for several reasons. This subsection discusses two scenarios where the developed proxy is beneficial to be deployed in practice. The first scenario is related to the proxy deployment in an official air pollution monitoring station. It is known that the pollutant ${\mathrm{O}}_3$ concentration differs between weekend and weekday, as well as day and night, due to the variation of traffic volumes in big cities [63]. An investigation of weekday and weekend differences in ${\mathrm{O}}_3$–${\mathrm{NO}}_{\mathrm{x}}$ levels is an important indicator to understand whether ${\mathrm{O}}_3$ has its origin in local photochemical production or in transport processes [54,64]. Figure 11 presents the average of the measured and estimated ${\mathrm{O}}_3$ concentration on weekday and weekend. It can be seen that lower ${\mathrm{NO}}_{\mathrm{x}}$ levels and higher ${\mathrm{O}}_3$ values at weekends than on weekdays took place in Jeddah. This phenomenon is common to be observed mainly within areas with an influence from urban emissions. This occurs due to weekly changes in emissions from human activities. The relationship of emission–concentration relationship at urban, suburban and rural sites is discussed in Stephens et al. [65]. Considering the significance of having both measurements of ${\mathrm{O}}_3$ and ${\mathrm{NO}}_{\mathrm{x}}$ concentration, the developed proxy performance on the ${\mathrm{O}}_3$ is evaluated on the concentration characteristic in the area as shown in Figure 11. It can be seen clearly that our developed estimated ${\mathrm{O}}_3$ concentration fits very well the average measurement data on weekday and weekend. This suggests that the developed ${\mathrm{O}}_3$ proxy is reliable and accurate as the substitution of ${\mathrm{O}}_3$ real measurement. Having the ${\mathrm{O}}_3$ virtual measurement is then beneficial to study the pollutant sources by reducing the cost and operator involvement.

The second scenario concerns the proxy deployment to complement low-cost sensor measurements in dense air pollution networks. Air quality networks in cities are usually sparse because air pollution is difficult to measure, instrumental and operational costs of official measurement stations are typically expensive [66]. On the other hand, air pollutant concentrations vary highly over both space and time [67]. Therefore, real-time and high-resolution measurements are then required that enable the mapping of air pollutants and air quality index in higher resolution. Over the last few years, the deployment of low cost sensors have been assessed for their viability in monitoring ambient air quality [68]. However, their accuracy and reliability are still challenging. For example, although testing low-cost ${\mathrm{O}}_3$ sensors demonstrates good performance under controlled labouratory conditions [69], their performances decline significantly under field conditions [70,71]. The performance degradation is mainly related to interference with other air pollutants as well as with meteorological conditions [72,73,74,75]. Therefore, it is also advantageous to substitute low-cost ${\mathrm{O}}_3$ sensors by ${\mathrm{O}}_3$ proxies. Although low-cost WS devices are also available, such as described in Pan et al. [76], but this variable may be neglected as the proxy input because the proxy performance is still reasonable well (see Table 1). The ${\mathrm{O}}_3$ proxy can then be deployed by taking inputs from the measurements of low-cost $\mathrm{NO}$/${\mathrm{NO}}_2$/${\mathrm{NO}}_{\mathrm{x}}$ sensors. The combination of low-cost sensors and virtual sensors is promising because virtual sensors are not affected directly by physical sensor failures and environmental conditions. However, the proxy accuracy might drift over time because there may be new measured data that has not been involved at all in the process of proxy development, which is known as extrapolation. This phenomenon can be detected through uncertainty quantification from the probabilistic ML based proxy and then the proxy can be retrained and updated accordingly.

5. Conclusions

This paper demonstrates the development of an air pollutant proxy based on data-driven methods. The deployment of air pollutant proxy is beneficial to replace missing data, to substitute a real measurement that are typically expensive and required operators and to be embedded in low-cost sensors as a virtual sensor. This work presents two main contributions in air pollutant proxy development: (1) mutual information (MI) for selecting appropriate inputs and (2) the proxy based on a probabilistic ML that is a Bayesian neural network (BNN).

The MI method demonstrates the effectiveness in finding other relevant variables to a pollutant to be modelled (as a proxy) where standard correlation method is unable to discover them. The proposed method guarantees to capture linear and nonlinear relationship among the variables. The most relevant variables as the proxy inputs are determined based on several highest MI scores obtained and based on proxy performance metrics as well as data loss monitoring. For practicality, the selected proxy inputs are then evaluated based on the number of instruments involved and their respective performance metrics. In this way, testing all combinations of measured variables can be avoided. A probabilistic ML method, named BNN, is then used to establish a pollutant proxy. The proposed method offers confidence interval for further analysis. The use of Bayesian method also prevents overfitting naturally due to the presence of regularisation and dropout method.

In particular, establishing an ${\mathrm{O}}_3$ concentration proxy is selected as a case study. The ${\mathrm{O}}_3$ and other variables’ databases are obtained from an extensive air pollution research campaign in Jeddah, Saudi Arabia. The MI application in the data results in the associated variables to ${\mathrm{O}}_3$ which have also been reported in other studies separately. The variables are ${\mathrm{NO}}_2$, ${\mathrm{NO}}_{\mathrm{x}}$, WS and TNX. The overall implementation of our proposed methodology on the case study leads to adequate performance in general and the obtained results are also considerably better in comparison with the previous attempt using the same dataset. As this is a general methodology, the strategy is also promising to be adopted to estimate any other types of air pollutant concentration.

Several future works can be continued from this approach. First, the proxy performance and accuracy can still be improved. The slight bias in the proposed proxy might take place due to the presence of autocorrelated effect in the data or the drifting in other measured variables. The former can be solved by applying dynamic ML models where they typically accommodate some previous steps of historical data in model inputs and outputs into the model. The latter can be checked by diagnosing the drift. For example, anomalies in the variables involved should be detected in the estimation given the absent of ground truth which can be performed using a predictive distribution analysis. If the drifting occurs, the remedy is then to develop an adaptive proxy which is capable to modify automatically its parameters based on the nature change in the data. Second, the capability of developed proxy has a potential to be enhanced for acting as a pollutant forecasting. Therefore, the proxy does not only estimate the current pollutant concentration, but also forecast it a couple of hours/days in advance, so better mitigation can be taken by policy makers. This can be done by applying MI between variables at different times for analysing and determining the dynamic relationships between air pollution variables. Then, the proxy can be upgraded into a dynamic model, such as Bayesian recurrent neural network (BRNN). In this case, a forecast proxy is fitted on newly available data prior to each prediction. Third, the developed proxy may not work well in other geographical locations. The proxy needs to be improved to ensure that it works robustly at other sites too. The involvement of Bayesian method in the proxy already allows the evaluation of the drifting effect which inform about the model performance through its uncertainty quantification. In addition, the proxy needs to be upgraded into a dynamic model as mentioned earlier. This makes the proxy becoming more adaptive because the model is updated when there is new data coming into the proxy. The involvement of additional inputs, such as meteorological variables, will also produce a generalised proxy, because these types of input distinguish different sites. However, the proxy needs to be trained at several different locations. Fourth, the extra measurements should be carried out in the same location using low-cost sensor devices next to the reference station. the developed proxy is then taken inputs from low-cost sensors’ measurements as inputs. In this way, we can further validate the usefulness of our proxy. Finally, as the developed method is generic, the proposed concept should be applied to the same/different types of air pollutant in same or different cities/regions.