Applying PCA to Deep Learning Forecasting Models for Predicting PM_2.5

Abstract

Fine particulate matter (PM_2.5) is one of the main air pollution problems that occur in major cities around the world. A country’s PM_2.5 can be affected not only by country factors but also by the neighboring country’s air quality factors. Therefore, forecasting PM_2.5 requires collecting data from outside the country as well as from within which is necessary for policies and plans. The data set of many variables with a relatively small number of observations can cause a dimensionality problem and limit the performance of the deep learning model. This study used daily data for five years in predicting PM_2.5 concentrations in eight Korean cities through deep learning models. PM_2.5 data of China were collected and used as input variables to solve the dimensionality problem using principal components analysis (PCA). The deep learning models used were a recurrent neural network (RNN), long short-term memory (LSTM), and bidirectional LSTM (BiLSTM). The performance of the models with and without PCA was compared using root-mean-square error (RMSE) and mean absolute error (MAE). As a result, the application of PCA in LSTM and BiLSTM, excluding the RNN, showed better performance: decreases of up to 16.6% and 33.3% in RMSE and MAE values. The results indicated that applying PCA in deep learning time series prediction can contribute to practical performance improvements, even with a small number of observations. It also provides a more accurate basis for the establishment of PM_2.5 reduction policy in the country.

1. Introduction

Fine particulate matter (PM_2.5) indicates particles with an aerodynamic diameter of 2.5 μm or less. It is not a specific chemical, such as sulfur oxides (${\mathrm{SO}}_x$) and nitrogen oxides (${\mathrm{NO}}_x$), but a mixture of particles of varying sizes, components, and shapes. Typical substances that form PM_2.5 include elemental carbon (EC), organic carbon (OC), ${\mathrm{NO}}_x$, volatile organic compounds (VOC), ozone (${\mathrm{O}}_3$), ammonia (${\mathrm{NH}}_3$), ${\mathrm{SO}}_x$, condensate particles, metal particles, mineral particles, etc. Because of its small size, it penetrates the body through the respiratory tract, causing inflammation or damaging organs [1]. The WHO considers PM_2.5 a major environmental risk factor that causes cardiovascular, respiratory, and various other cancers [2]. Figure 1 shows the effects of PM_2.5 on the body [3].

Figure 1. Effects of fine particulate matter (PM_2.5) on the body. Source: French National Health Agency, InVS (Institut de veille sanitaire), European Environment Agency, and AFP.

Korea’s PM_2.5 concentration was the highest among the 37 OECD (Organization for Economic Co-operation and Development) countries in 2019 [4], and studies have shown that it has a negative effect on people’s health. Han et al. [5] stated that 1763 early deaths in Seoul in 2015 were closely related to PM_2.5. Hwang et al. [6] explained that, when the average annual concentration of PM_2.5 in Seoul increases by $10\mathsf{\mu }\mathrm{g}/{\mathrm{m}}^3$, the risk of death over 65 years increases by 13.9%. This is in line with the major causes of death for Koreans in 2019. Statistics Korea shows that cancer (158.2 deaths per 100,000 people), cardiovascular diseases (60.4 deaths per 100,000 people), and pneumonia (45.1 deaths per 100,000 people) are the three major causes of death [7]. This suggests that PM_2.5 is highly correlated to the main cause of death for Koreans.

The Korean government is making great efforts to reduce PM_2.5 concentration to protect people’s health. The government has divided the crisis into three stages according to the current status and prediction of PM_2.5 concentration and has devised a manual for local governments for each stage of action. The government also aims to reduce the annual average concentration of PM_2.5 by 35% compared to 2016 by establishing a five-year plan for PM_2.5 concentration reduction. To achieve this purpose, the government selected 15 major tasks by evaluating its potential reduction, cost effectiveness, linkage with other policies, and social impact. These tasks are implemented by each local government [8].

Table 1 shows Korea’s crisis stage standard for PM_2.5 concentration, which reflects the concentration of PM_2.5 in the current period and future forecast values. It suggests that the accurate prediction of PM_2.5 concentration is needed in the short and long terms. In this regard, several studies have conducted air quality prediction using deep learning methods with domestic data (wind speed, NO₂, SO₂, temperature, etc.) in Korea, and new deep learning models have been developed to show high performance in air quality prediction [9,10]. However, foreign factors should also be considered in predicting PM_2.5 concentration in Korea, as the concentration of PM_2.5 in the Shandong region of China is also found to affect Korea’s PM_2.5 concentration [11]. However, as China’s past PM_2.5 concentration data are composed of daily data, Korea’s data should also be organized on a daily basis for deep learning PM_2.5 prediction. This data composition can cause a “curse of dimensionality” due to the small number of observations compared to variables, which can reduce the performance of the model.

Table 1. Crisis stage standard.

Crisis Stages	Criteria	Main Contents
Stage 1	150 μg/m3 for 2 h or longer + 75 μg/m3 for the following day	Strengthening the current system
Stage 2	200 μg/m3 for 2 h or longer + 150 μg/m3 for the following day	Strengthening public sector measures
Stage 3	400 μg/m3 for 2 h or longer + 200 μg/m3 for the following day	Strengthening private sector measures/disaster response

This study aims to show that the application of principal component analysis (PCA) in the deep learning time series prediction models for PM_2.5—a recurrent neural network (RNN), long short-term memory (LSTM), and bidirectional LSTM (BiLSTM)—can result in better performance by comparing the root-mean-square error (RMSE) and mean absolute error (MAE) with the same models without PCA application.

2. Previous Research

Several studies have shown the association of PM_2.5 with lung and cardiovascular disease (CVD). Wang et al. [12] reported that CVD is the one of the main mortality factors of elder people. It was found that the ambient PM_2.5 concentration is related to several CVDs by linking PM_2.5 exposure and CVD based on multiple pathophysiological mechanisms. César et al. [13] showed that the exposure to PM_2.5 can cause hospitalizations for pneumonia and asthma in children younger than 10 years of age through an ecological study of time series and a generalized additive model of Poisson regression. Kim et al. [14] reported associations of short-term PM_2.5 exposure with acute upper respiratory infection and bronchitis among children aged 0–4 years through a difference-in-differences approach generalized to multiple spatial units (regions) and time periods (day) with distributed lag non-linear models. Vinikoor-Imler et al. [15] studied the relationship between PM_2.5 concentration, lung cancer incidence, and mortality by linear regression and concluded that there is a possibility of an association between them. Choe et al. [16] reported that the effect of changes in PM_2.5 emissions on changes in internal visits and hospitalization probabilities due to respiratory diseases was estimated through Probit and Tobit models. If PM_2.5 emissions change by 1%, the probability of visitation due to respiratory diseases increases from 0.755% to 1.216%, and the probability of hospitalization increases from 0.150% to 0.197%.

The need for ${\mathrm{PM}}_x$ prediction research is emerging, and various studies are underway on ${\mathrm{PM}}_x$ prediction. Zev Ross et al. [17] developed the land use regression model to predict PM_2.5 in New York City and showed that urbanization factors such as traffic volume and population density have a high explanation in predicting PM_2.5. Rob Beelen et al. [18] compared the performance of ordinary kriging, universal kriging, and regression mapping in developing EU-wide maps of air pollution and showed that universal kriging performs better in mapping ${\mathrm{NO}}_2$, ${\mathrm{PM}}_{10}$, and ${\mathrm{O}}_3$. Vikas Singh et al. [19] suggested a cokriging-based approach and interpolated ${\mathrm{PM}}_{10}$ in areas not observed in the network in ${\mathrm{PM}}_{10}$ monitoring based on the suggested method with secondary variable from the results of a deterministic chemical transport model (CTM) simulation. And the results showed that the proposed method provides flexibility in collecting ultrafine dust data.

Other studies have shown examples of predicting PM_2.5 through machine learning and deep learning. Zhao et al. [20] predicted the PM_2.5 contamination of stations in Beijing using long short-term memory—fully connected (LSTM-FC), LSTM, and an artificial neural network (ANN) with historical air quality data, meteorological data, weather forecast data, and the day of the week data. They showed that the LSTM-FC model outperforms LSTM and the ANN, with MAE = 23.97–50.13 and RMSE = 35.82–69.84 over 48 h. Karimian et al. [21] also predicted Tehran’s PM_2.5 concentration by implementing multiple additive regression trees (MARTs), a deep feedforward neural network (DFNN), and a new hybrid model LSTM with meteorological data (temperature, surface-level pressure, relative humidity, etc.). The best model in this research was LSTM in 12, 24, and 48 h prediction, with RMSE = 7.03–11.73 μg/m³ and MAE = 5.59–8.41 μg/m³. Qadeer et al. [22] used XGBoost (XGB), the light gradient boosting machine (LGBM), the gated recurrent unit (GRU), convolutional neural network–LSTM (CNNLSTM), BiLSTM, and LSTM to predict PM_2.5 concentration of eight sites in Seoul and Gwangju with community multiscale air quality (CMAQ) data. The result showed that LSTM performs best, with MAE = 3.5847 μg/m³, RMSE = 4.8292 μg/m³, R = 0.8989, and IA = 0.9368 of the mean in all sites.

The RNN, LSTM, and BiLSTM models were used in this study, because previous studies have shown that the deep learning sequence model performs better in prediction. The local weather and air quality data were used to predict PM_2.5, as shown in previous studies, and used as predictive input variables. The regional data of China are also used as predictive input variables, which were found to affect PM_2.5 in Korea.

3. Data

3.1. Spatial Area

Figure 2 shows the spatial range of the research. A total of eight cities in Korea were selected for analysis. Of the eight cities, six are metropolitan cities (Busan, Daejeon, Daegu, Gwangju, Incheon, and Ulsan) representing each province, one is the capital city (Seoul), and one is the most populous city (Wonju) in the province without a metropolitan city. In each city, daily air quality data (PM_2.5, SO₂, O₃, NO₂, and CO) [23] and meteorological data (temperature, wind speed, wind direction, humidity, precipitation, etc.) [24] were collected in consideration of the internal factors of PM_2.5 generation. Air quality data were collected within 5 km of each city’s meteorological data observatory.

Figure 2. Spatial range of the research.

Figure 3 shows that Korea is mainly a country with north and west winds. As a result, the air quality of Korea can be directly and indirectly affected by the air quality of China, a country located in the west and north. Figure 4 [25] also shows the concentration of PM_2.5 in Korea and China at the same time before and after the outbreak of COVID-19. According to Bao et al. [26], it can be seen that the lockdown of Chinese factories after the COVID-19 outbreak actually improved the Chinese air quality. Considering this, with the direction of the wind in Korea, we can see that the air quality of Korea is highly affected by the air quality in China. Accordingly, daily PM_2.5 concentrations in 55 areas in China close to Korea were selected as input variables in this study, including the PM_2.5 concentration in Shandong province, which was found to increase PM_2.5 concentration in Korea.

Figure 3. The wind direction frequency of Korea’s selected cities in 2019.

Figure 4. PM_2.5 distribution maps (before and after the COVID-19 outbreak: 2019 vs. 2020).

3.2. Data Preprocessing

All variables have a time range from 1 January 2015 to 31 December 2019 and are collected as daily data. There are missing values in some variables, and these missing values were processed by the exponentially weighted moving average (EWMA) using the imputeTS package of the R software [27]. The EWMA gives higher weights to the latest data, reducing the weight of older values, and the formula for EWMA imputation suggested by Hunter [28] is as follows:

$${\widehat{S}}_t={\widehat{S}}_{t-1}+\alpha e_{t-1}$$

(1)

$$={\widehat{S}}_{t-1}+\alpha \left(S_{t-1}-{\widehat{S}}_{t-1}\right)$$

(2)

$$=\alpha S_{t-1}+\left(1-\alpha \right){\widehat{S}}_{t-1}$$

(3)

$$⋮$$

$$=\alpha {\displaystyle \sum }_{k=1}^{t-2}{\left(1-\alpha \right)}^{k-1}{S}_{t-k}+{\left(1-\alpha \right)}^{t-2}{S}_2$$

(4)

$$*\alpha =\frac{2}{n+1},n=MovingAveragePeriod,k\in \left\{1,2,\dots \right\},t\ge 2$$

(5)

${\widehat{S}}_t$ is the predicted value at time t, $S_t$ is the observed value at time t, $e_t$ is the observed error at time t, and $\alpha$ is a constant value called the weight from zero to one. The higher the $\alpha$ value is, the less it reflects past data.

Figure 5 and Figure A1, Figure A2 and Figure A3 show the concentration of PM_2.5 in China (Figure 5a), Seoul (Figure 5b) with air quality, and the meteorological data of Seoul (Figure A1, Figure A2 and Figure A3). Each variable shows the values in a different range due to the differences in units of measurement and the characteristics within the region. In the case of Chinese data, the concentration of PM_2.5 in each city over time seems to be constant, but some cities have outliers. If one variable has a relatively greater value, or a wider range of values than the others, in the composition of the data, it can result in a significant impact on the predicted value, regardless of the predictive importance of the variable.

Figure 5. Visualization of China’s and Seoul’s air quality data set.

To solve these problems, the scope of the variables should be adjusted through normalization. In this study, maximum–minimum normalization was carried out to every data of each city as shown in the following equation:

$$NormalizedVariable\prime svalue=\frac{Variabl{e}^{\prime }sOriginalvalue-Variabl{e}^{\prime }sMinimumvalue}{Variabl{e}^{\prime }sMaximumvalue-Variabl{e}^{\prime }sMinimumvalue}$$

(6)

Because the wind direction data were collected as 16 cardinal points, these are labels encoded to transform direction data into numerical data.

3.3. Variable Correlation Analysis

As mentioned above, the prediction target of this study is the concentration of PM_2.5. The efficiency of the forecast results in deep learning, and machine learning depends on the correlation between the dependent and the independent variables. It is important to add variables with a strong negative or positive correlation between the dependent variable and the independent variable. In addition, the results of correlation are necessary for data analysis because they provide a basis for determining the influence of each independent variable on a dependent variable. In this study, the Pearson correlation coefficient was calculated, which is expressed as the covariance and standard deviation of the variables, as shown in the following equations in the case of observation vector $\mathrm{X}=\left(X_1,X_2,X_3\dots ,X_n\right)$:

$$CorrelationMatrix$$

$$=\left[\begin{array}{cc}\frac{{\sum }^{\text{}}\left(X_1-\overline{X_1}\right)\left(X_1-\overline{X_1}\right)}{\sqrt{{\sum }^{\text{}}{\left(X_1-\overline{X_1}\right)}^2}\sqrt{{\sum }^{\text{}}{\left(X_1-\overline{X_1}\right)}^2}}\frac{{\sum }^{\text{}}\left(X_1-\overline{X_1}\right)\left(X_2-\overline{X_2}\right)}{\sqrt{{\sum }^{\text{}}{\left(X_1-\overline{X_1}\right)}^2}\sqrt{{\sum }^{\text{}}{\left(X_2-\overline{X_2}\right)}^2}}& \cdots \frac{{\sum }^{\text{}}\left(X_1-\overline{X_1}\right)\left(X_n-\overline{X_n}\right)}{\sqrt{{\sum }^{\text{}}{\left(X_1-\overline{X_1}\right)}^2}\sqrt{{\sum }^{\text{}}{\left(X_n-\overline{X_n}\right)}^2}}\\ ⋮& ⋮\\ ⋮& ⋮\\ \frac{{\sum }^{\text{}}\left(X_n-\overline{X_n}\right)\left(X_1-\overline{X_1}\right)}{\sqrt{{\sum }^{\text{}}{\left(X_{\mathrm{n}}-\overline{X_{\mathrm{n}}}\right)}^2}\sqrt{{\sum }^{\text{}}{\left(X_1-\overline{X_1}\right)}^2}}\frac{{\sum }^{\text{}}\left(X_{\mathrm{n}}-\overline{X_{\mathrm{n}}}\right)\left(X_2-\overline{X_2}\right)}{\sqrt{{\sum }^{\text{}}{\left(X_{\mathrm{n}}-\overline{X_{\mathrm{n}}}\right)}^2}\sqrt{{\sum }^{\text{}}{\left(X_2-\overline{X_2}\right)}^2}}& \cdots \frac{{\sum }^{\text{}}\left(X_n-\overline{X_n}\right)\left(X_n-\overline{X_n}\right)}{\sqrt{{\sum }^{\text{}}{\left(X_n-\overline{X_n}\right)}^2}\sqrt{{\sum }^{\text{}}{\left(X_n-\overline{X_n}\right)}^2}}\end{array}\right]$$

(7)

$$=\left[\begin{array}{cc}\begin{array}{cc}\frac{Cov\left(X_1,X_1\right)}{\sqrt{Var\left(X_1\right)}\sqrt{Var\left(X_1\right)}}& \begin{array}{cc}\frac{Cov\left(X_1,X_2\right)}{\sqrt{Var\left(X_1\right)}\sqrt{Var\left(X_2\right)}}& \end{array}\end{array}& \cdots \frac{Cov\left(X_1,X_n\right)}{\sqrt{Var\left(X_1\right)}\sqrt{Var\left(X_n\right)}}\\ ⋮& ⋮\\ ⋮& ⋮\\ \begin{array}{cc}\frac{Cov\left(X_n,X_1\right)}{\sqrt{Var\left(X_n\right)}\sqrt{Var\left(X_1\right)}}& \frac{Cov\left(X_n,X_2\right)}{\sqrt{Var\left(X_n\right)}\sqrt{Var\left(X_2\right)}}\end{array}& \frac{Cov\left(X_n,X_n\right)}{\sqrt{Var\left(X_n\right)}\sqrt{Var\left(X_n\right)}}\end{array}\right]$$

(8)

Each element in the correlation matrix has a value between −1 and 1, showing that a value greater than 0 is a positive correlation and a value less than 0 is a negative correlation. The correlation matrix is symmetric, and all of the diagonal elements of the matrix have a value of 1 considering $Cov\left(X_i,X_i\right)=Var\left(X_i\right)$, $i\in \left\{1,2,\dots n\right\}$.

Figure 6 is a visualization of the correlation between ${\mathrm{PM}}_{2.5}$ concentrations and the highest eight factors inside Seoul, Korea. Appendix A Table A2, Table A3, Table A4, Table A5, Table A6, Table A7, Table A8 and Table A9 show the correlation between PM_2.5 concentrations and the meteorological air quality factors of each city in Korea. Overall, the factors that have a strong positive correlation with PM_2.5 are air quality factors except for O₃. PM_2.5 also appears to have a positive correlation with local air pressure (LAP), sea-level pressure (SP), wind direction, and relative humidity. Conversely, temperature, wind speed, O₃, wind flow sum (wind flow sum refers to the distance that the air flows, and the Korea Meteorological Administration produces a day-to-day wind flow sum (24 h wind flow sum).), and daily precipitation were found to have a negative correlation with PM_2.5 concentrations. However, the variables that have a relatively weak correlation with PM_2.5 changed the sign of the correlation depending on the region.

Figure 6. Correlation between the highest eight factors and PM_2.5 concentrations in Seoul, Korea.

Figure 7 shows an origin–destination map of PM_2.5 correlations between Chinese [29] and Korean cities. The correlations between PM_2.5 concentrations in each Chinese city and PM_2.5 concentrations in each Korean city vary, but as shown in Table 2, an overall correlation between 0.13 and 0.55 is shown. Comparing this with the factors inside the Korean cities, we can see that the PM_2.5 concentration of each city in China is as much related with the PM_2.5 concentration in Korea as the data of air quality inside the city. This suggests that China’s PM_2.5 concentration could be an important independent variable in predicting PM_2.5 concentrations in Korea.

Figure 7. Origin–destination map of PM_2.5 correlations between Chinese cities and Korean cities.

Table 2. Correlation range from Chinese cities to Korean cities.

Cities	Minimum	Maximum	Cities	Minimum	Maximum
Seoul	0.1993	0.4994	Busan	0.1320	0.5084
Gwangju	0.1446	0.4035	Ulsan	0.1419	0.5394
Daegu	0.1813	0.5087	Wonju	0.1824	0.5550
Daejeon	0.1839	0.5087	Incheon	0.2556	0.5415

4. Analytical Methods

4.1. PCA

PCA reduces dimensions by linear combinations of variables with high explanatory power of the overall data variability, explaining variation in high-dimension data in low dimensions. Vectors with p variables can have total p principal components, and the principal components of vector $\mathit{x}\left(1\times p\right)$, whose covariance matrix is $\mathit{\Sigma }\left(p\times p\right)$, can be generated as follows:

$$PC={\mathit{a}}^T\mathit{x}={\mathit{a}}_{\mathbf{1}}{\mathit{x}}_{\mathbf{1}}+{\mathit{a}}_{\mathbf{2}}{\mathit{x}}_{\mathbf{2}}+\cdots +{\mathit{a}}_{\mathit{p}}{\mathit{x}}_{\mathit{p}}$$

(9)

$$Var\left({\mathit{a}}^T\mathit{x}\right)={\mathit{a}}^{T}Var\left(\mathit{x}\right)\mathit{a}={\mathit{a}}^T\mathit{\Sigma }\mathit{a}$$

(10)

$$ℒ={\mathit{a}}^T\mathit{\Sigma }\mathit{a}-\lambda \left({\mathit{a}}^T\mathit{a}-1\right)$$

(11)

$$\frac{\partial ℒ}{\partial \mathit{a}}=2\mathit{\Sigma }\mathit{a}-2\lambda \mathit{a}=\mathbf{0}$$

(12)

$$\mathit{\Sigma }\mathit{a}=\lambda \mathit{a}$$

(13)

$$Var\left(PC\right)={\mathit{a}}^T\mathit{\Sigma }\mathit{a}={\mathit{a}}^T\left(\lambda \mathit{a}\right)=\lambda$$

(14)

$$P{C}_i={\mathit{a}}_i^T\mathit{x}={\mathit{a}}_{\mathit{i}\mathbf{1}}{\mathit{x}}_{\mathbf{1}}+{\mathit{a}}_{\mathit{i}\mathbf{2}}{\mathit{x}}_{\mathbf{2}}+\cdots +{\mathit{a}}_{\mathit{i}\mathit{p}}{\mathit{x}}_{\mathit{p}}$$

(15)

$$Var\left(P{C}_i\right)={\lambda }_{\mathit{i}}$$

(16)

Because the principal component is a linear combination of $\mathrm{X}$, it can be expressed as Equation (9), and the variance of this linear combination can be expressed as Equation (10). The PCA has to preserve the variance of the original data as much as possible, so Equation (10) should also be maximized. Therefore, the method of generating principal components can be transformed into the problem of obtaining $\mathit{a}\left(p\times 1\right)$, which maximizes ${\mathit{a}}^T\mathit{\Sigma }\mathit{a}$ under the condition ${\mathit{a}}^T\mathit{a}=\mathbf{1}$. Equation (11) was derived by applying Lagrange’s multiplier method to Equation (10). Equation (13) was made by Equation (12), which partially differentiates Equation (11) by $\mathit{a}$. Equation (13) shows that $\lambda$ is the eigenvalue of $\mathit{\Sigma }$, and $\mathit{a}$ is the eigenvector of $\mathit{\Sigma }$. As a result, a linear combination that maximizes Equation (10), i.e., the principal component, can be expressed as Equation (9). In addition, Equation (10), which is the variance of the principal component, can be expressed as $\lambda$ under the condition ${\mathit{a}}^T\mathit{a}=\mathbf{1}$. Therefore, in vectors with p variables, the i-th principal component is Equation (15), and the variance is Equation (16). Subsequently, the number of principal components is selected for convenience by the principal components where the sum of the principal components is more than 80% to 90% of the total variance. For example, the number of principal components i has to be selected out of principal components p. Equation (17) has to produce results of more than 80% to 90%:

$$\frac{{\lambda }_{\mathbf{1}}+{\lambda }_{\mathbf{2}}+\cdots +{\lambda }_{\mathit{i}}}{{\lambda }_{\mathbf{1}}+{\lambda }_{\mathbf{2}}+{\lambda }_{\mathbf{3}}+{\lambda }_{\mathbf{4}}+\cdots +{\lambda }_{\mathit{p}}}$$

(17)

4.2. RNN

The RNN is a deep learning model for processing sequence data, such as stock charts [30], music [31], and natural language processes [32]. It remembers the state entered from the previous time point (t − 1) through the hidden layer and passes the hidden layer state at that specific time point (t) to the next time point (t + 1). That is, the status at the previous time point affects the state at the present time point, and the state at the present time point affects the status at the next time point. This procedure is repeated until result values becomes optimized; hence the name “recurrent neural network.”

$$h_{t-1}=tanh\left(W_{hh}{h}_{t-2}+W_{xh}{x}_{t-1}+b_h\right)$$

(18)

$$h_t=tanh\left(W_{hh}{h}_{t-1}+W_{xh}{x}_t+b_h\right)$$

(19)

$$\widehat{y_t}=W_{hy}{h}_t+b_y$$

(20)

$$L_t=MSE=\frac{\sum {\left(y_t-{\widehat{y}}_t\right)}^2}{n}$$

(21)

Figure 8b is the unrolled and inner structure of Figure 8a. In Equations (18)–(20), $x_t$ is an input, and $h_t$ is a hidden state at time t. $W_{ij}$ is the weight from layer i to layer j, and $b_i$ is the bias in each layer. In Equation (21), $L_t$ is the loss at time t, and $y_t$ and $\widehat{y_t}$ are the actual and predicted values, respectively, at time point t.

Figure 8. Internal structure of the recurrent neural network (RNN).

The RNN model shares the weights and biases at all time points and circulates the input data to output the results. Model training is repeated until the loss value is minimized by gradient descending in the loss function, with information of specific previous time steps. At the same time, the weight is updated to find the optimum value. This is called backpropagation through time (BPTT) and in an RNN can be expressed as follows [33]:

$$Updated{W}_{xh}=Existing{W}_{xh}-\eta {\displaystyle \sum }_{t=1}^n{\displaystyle \sum }_{k=0}^n\frac{\partial L_t}{\partial {\widehat{y}}_t}\frac{\partial {\widehat{y}}_t}{\partial h_t}\frac{\partial h_t}{\partial h_k}\frac{\partial h_k}{\partial W_{xh}}$$

(22)

$$Updated{W}_{hh}=Existing{W}_{hh}-\eta {\displaystyle \sum }_{t=1}^n{\displaystyle \sum }_{k=0}^n\frac{\partial L_t}{\partial {\widehat{y}}_t}\frac{\partial {\widehat{y}}_t}{\partial h_t}\frac{\partial h_t}{\partial h_k}\frac{\partial h_k}{\partial W_{hh}}$$

(23)

$$Updated{W}_{hy}=Existing{W}_{hy}-\eta {\displaystyle \sum }_{t=1}^n\frac{\partial L_t}{\partial W_{hy}}$$

(24)

$$\ast \eta =learningrate\left[0,1\right]$$

4.3. LSTM and BiLSTM

In an RNN, tanh is used as an activation function to train the model in a non-linear way. However, there is a long-term dependency problem caused by a “vanishing gradient” problem in the RNN’s BPTT, in which the gradient (weights update rate) disappears as the value (derivative value of the tanh function with respect to $h_t$) less than 1 continues to multiply. Thus, the state of a relatively distant past time point has almost no effect on an output of the present time point. As a result, the model relies only on short-term data and has a limit in achieving the best performance. To solve this problem, Hochreiter et al. [34] suggested the LSTM model. Figure 9 shows the internal structure of LSTM and its process.

Figure 9. Internal structure of long short-term memory (LSTM).

LSTM is the model in which forgetting and memory ($f_t$), the input ($i_t$), the inner cell state candidate $\left(\widetilde{C_t}\right)$, the conveying and inner cell state at time point t ($C_t$), and the output ($o_t$) are added to the RNN model. Especially, $C_{t }$, which penetrates all time points, greatly contributes to solving the long-term dependency problem. The order of each part and the internal algorithm can be explained by the following process:

$$f_t=\sigma \left(W_{xh\left(f\right)}{x}_t+W_{hh\left(f\right)}{h}_{t-1}+b_{h\left(f\right)}\right)$$

(25)

$$i_t=\sigma \left(W_{xh\left(i\right)}{x}_t+W_{hh\left(i\right)}{h}_{t-1}+b_{h\left(i\right)}\right)$$

(26)

$$\widetilde{C_t}=tanh\left(W_{xh\left(\widetilde{C_t}\right)}{x}_t+W_{hh\left(\widetilde{C_t}\right)}{h}_{t-1}+b_{h\left(\widetilde{C_t}\right)}\right)$$

(27)

$$C_t=f_t\odot C_{t-1}+i_t\odot \widetilde{C_t}$$

(28)

$$o_t=\sigma \left(W_{xh\left(o\right)}{x}_t+W_{hh\left(o\right)}{h}_{t-1}+b_{h\left(o\right)}\right)$$

(29)

$$h_t=o_t\odot tanh\left(C_t\right)$$

(30)

$$\ast \odot =Hadamardproduct,\sigma =sigmoidfunction=\frac{1}{1+e^{-x}}$$

Equation (25), output of the forget gate, determines whether the historical state is forgotten by the combination of $x_t$ and $h_{t-1}$. The output value of this step is converted to a number between 0 and 1 by the sigmoid function and multiplied by $C_{t-1}$ (memory of past data, i.e., historical state) to determine how much past data to preserve or forget. A value of 0 indicates forgetfulness, and 1 indicates memorization of past data. Equations (26) and (27) are involved in the storage of the inner cell state of time point t. Equation (26), output of the input gate, determines how much data of time point t are memorized. In other words, it has a value between 0 and 1, indicating the degree of memorizing for the new information. At the same time, Equation (27) generates the inner cell state candidate of time point t. Equation (28) generates the new cell state at time point t and passes it on to the LSTM cell at the next time point (t + 1). In other words, LSTM solves the RNN’s long-term dependency problem by adjusting the memorization and forgetfulness of the past and presents the state through Equations (25)–(28). In the end, the output is decided by Equations (29) and (30). Equation (29), output of the output gate, decides which part of the new cell state will become output. A value of the new cell status is converted through the tangent function and calculated with the result value of Equation (29) to produce the final output of time point t, as shown in Equation (30).

BiLSTM is a variant of the bidirectional RNN proposed by Schuster et al. [35]. Figure 10 shows an example of applying a bidirectional way to sentence learning. If (A) is taught in the model and “went” is set as the target, (B) predicts in a forward way and (C) predicts in both a forward and a backward way. If LSTM uses a historical state to predict the value of time point t, bidirectional LSTM predicts the value of time point t by adding an LSTM layer that reads data from a future state. The computations within the model are the same as those of LSTM, and LSTM and BiLSTM update their weights in the training model as an RNN [36].

Figure 10. Internal structure of bidirectional long short-term memory (BiLSTM) and an example.

4.4. Evaluation Model Performance

In this study, MAE (Mean Absolute Error) and RMSE (Root Mean Square Error) were used as evaluation indicators to compare the performance of each model with and without PCA application. The calculations of each indicator are expressed as follows:

$$MAE=\frac{1}{n}{\displaystyle \sum }_{t=1}^n\left|y_t-\widehat{y_t}\right|$$

(31)

$$RMSE=\sqrt{\frac{1}{n}{\displaystyle \sum }_{t=1}^n{\left(y_t-\widehat{y_t}\right)}^2}$$

(32)

4.5. Workflow

The flow of this study is divided into four stages: data collection, data preprocessing, prediction, and evaluation (Figure 11). The application of PCA is used in the data preprocessing stage, aiming to reduce the number of variables and increase the performance of model predictions. Thus, the data preprocessing stage was divided into two cases. Case 1 was set as a prediction without a PCA application, and Case 2 was set as a prediction with the PCA application. Afterwards, each case will be compared using evaluation indicators (MAE and RMSE).

Figure 11. Workflow of the PCA application deep learning model for predicting PM_2.5.

5. Results

5.1. PC Selection

For each city, PCA was performed on the input variables, except the dependent variable, PM_2.5. The variance of each city’s data was explained by a relatively small number of principal components, which resulted in the selection of five principal components in all cities. This reduced the number of input variables to about 1/16. Table A10, Table A11, Table A12, Table A13, Table A14, Table A15, Table A16 and Table A17 show the results of the PCA of each city, and Table 3 shows how much the five principal components describe the overall variation of each city.

Table 3. The ratio of variance explained by five principal components in each city.

Cities	Cumulative Variance	Cities	Cumulative Variance
Seoul	0.9631(=96.31%)	Busan	0.98102(=98.102%)
Gwangju	0.9553(=95.53%)	Ulsan	0.9655(=96.55%)
Daegu	0.9770(=97.70%)	Wonju	0.9366(=93.66%)
Daejeon	0.9539(=95.39%)	Incheon	0.98123(=98.123%)

5.2. Setup and Case Comparison

China’s daily PM_2.5 concentration and Korea’s air quality and meteorological data were collected from 1 January 2015 to 31 December 2019 to predict the concentration of PM_2.5 in eight Korean cities. In total, 85% of the collected data were allocated to the train set and 15% to the test set. In the aspect of details in models, the three models have 256 units in the layer, a tanh activation function, 200 epochs, a batch size of 64 and an adaptive moment estimation (ADAM) optimizer [37]. To avoid overfitting, 30% of the train set was designated as a validation set, and a 30% dropout regulation was used between the input layer and the output layer. Additionally, in model learning, earlystopping, one of the callback functions of Keras, was applied to stop learning in the epoch when optimal learning had achieved 200 epochs.

Figure 12 shows the predicted and actual values of PM_2.5 for each case and model in Seoul. Figure A4, Figure A5, Figure A6, Figure A7, Figure A8, Figure A9 and Figure A10 show the PM_2.5 concentration prediction of each city except for Seoul. Unlike LSTM and BiLSTM, the RNN appears to have outputted average values for all time periods and shows relatively low predictive power in both Case 1 and Case 2. The RNN without PCA seems to follow the trend more and to show relatively higher performance than the RNN with PCA. However, although there are differences between cities, LSTM and BiLSTM show that they follow the trend relatively well, regardless of whether PCA is applied or not. Furthermore, it can be seen that PCA application in all cities corrects the difference between the predicted and actual values that exists if PCA is not applied. It also appears to have produced more accurate results in predicting peak values.

Figure 12. The PM_2.5 prediction in Seoul by two cases.

Table 4 and Table 5 are numerical representations of these visual results. As noted above, it is understood that the reduction in dimension in all cities leads to a relatively low performance in the RNN, except for Daegu in terms of the MAE. This means that, for RNNs, reducing variables does not help model learning; rather, providing a high amount of information in a short period of time can lead to a better performance, depending on the feature of the model that relies on short-term information. Instead of an RNN, which lacks overall accuracy, the results that should be considered are those of LSTM and BiLSTM. Unlike the RNN, PCA application to LSTM and BiLSTM showed better results in RMSE and MAE evaluation, similar to the visual results. The order of cities with high performance is as follows: Busan > Daejeon > Gwangju > Daegu > Seoul > Ulsan > Wonju > Incheon, while the order of cities with high improvement in MAE and RMSE is as follows: Busan > Incheon > Gwangju > Seoul > Ulsan > Daegu > Daejeon > Wonju. LSTM showed high performance in Daejeon, Daegu, and Busan, while BiLSTM showed higher performance in the rest of the cities.

Table 4. Evaluation results from PM_2.5 prediction in each Korean city (Case 1).

City	Model	RMSE	MAE	City	Model	RMSE	MAE
Seoul	RNN	9.730	7.328	Gwangju	RNN	9.002	7.472
	LSTM	8.020	6.374		LSTM	7.7415	5.797
	BiLSTM	8.101	6.168		BiLSTM	8.300	6.590
Daegu	RNN	10.171	8.110	Busan	RNN	8.410	7.224
	LSTM	7.654	6.223		LSTM	7.770	6.504
	BiLSTM	7.707	6.193		BiLSTM	7.897	6.578
Daejeon	RNN	9.361	7.497	Ulsan	RNN	10.558	8.988
	LSTM	7.042	5.753		LSTM	8.660	6.959
	BiLSTM	7.231	5.927		BiLSTM	8.383	6.772
Wonju	RNN	11.603	9.208	Incheon	RNN	13.686	11.408
	LSTM	8.718	6.520		LSTM	11.900	9.828
	BiLSTM	8.459	6.251		BiLSTM	10.393	8.285

Table 5. Evaluation results from PM_2.5 prediction in each Korean city (Case 2).

City	Model	RMSE	MAE	City	Model	RMSE	MAE
Seoul	RNN	11.680 (20%↑)	9.310 (27%↑)	Gwangju	RNN	9.492 (5.4%↑)	7.746 (3.7%↑)
	LSTM	7.667 (4.6%↓)	5.455 (16.8%↓)		LSTM	7.148 (8.3%↓)	5.541 (4.6%↓)
	BiLSTM	7.567 (7.1%↓)	5.368 (14.9%↓)		BiLSTM	7.110 (16.7%↓)	5.455 (20.8%↓)
Daegu	RNN	10.208 (0.4%↑)	7.824 (3.5%↓)	Busan	RNN	9.924 (18%↑)	8.316 (15.1%↑)
	LSTM	7.491 (2.2%↓)	5.664 (9.9%↓)		LSTM	6.668 (16.5%↓)	4.881 (33.3%↓)
	BiLSTM	7.552 (2.1%↓)	5.703 (8.6%↓)		BiLSTM	6.779 (16.5%↓)	4.999 (31.6%↓)
Daejeon	RNN	9.602 (2.6%↑)	7.824 (4.4%↑)	Ulsan	RNN	11.160 (5.7%↑)	9.389 (4.5%↑)
	LSTM	6.967 (1.1%↓)	5.374 (7.1%↓)		LSTM	8.021 (8%↓)	6.251 (11.3%↓)
	BiLSTM	7.098 (1.9%↓)	5.537 (7%↓)		BiLSTM	7.871 (6.5%↓)	5.993 (13%↓)
Wonju	RNN	12.132 (4.6%↑)	9.758 (6%↑)	Incheon	RNN	14.744 (7.7%↑)	12.427 (8.9%↑)
	LSTM	8.424 (3.5%↓)	6.251 (4.3%↓)		LSTM	10.205 (16.6%↓)	8.000 (22.9%↓)
	BiLSTM	8.345 (1.4%↓)	6.137 (1.9%↓)		BiLSTM	9.709 (7%↓)	7.354 (12.7%↓)

The difference in performance and performance improvements city by city makes it worthwhile to consider which characteristics of each city would cause regional differences in the performance of the same model, and which model would perform better depending on regional characteristics. To do so, it is expected that such studies require multidisciplinary considerations.

6. Conclusions

Performance degradation due to the curse of dimensionality can occur in deep learning and machine learning. We proposed a PCA-applied model to solve this problem, and through performance comparison with a non-PCA model, we showed that PCA applications produce better results in deep learning time series prediction. Such a performance improvement technique can be a way to increase the efficiency of the government system by providing better forecasts as a basis for issuing crisis alerts and establishing air pollution reduction policies in the future.

As the correlation analysis shows, the concentration of PM_2.5 in China appears to have positive correlations with the concentration of PM_2.5 in China, indicating that we have to consider China’s air pollution factors in predicting the concentration of PM_2.5 in Korea. It suggests that there is a justification for the setup of real-time air pollution databases between the two countries from the ongoing joint research between Korea and China [38].

However, while PCA applications can improve model performance, the results show relatively weak predictions on predicting the minimum and maximum concentration PM_2.5 for each city. It seems to be a problem due to a small number of observations (daily observations, not hourly observations). It is expected that future joint cross-border research will result in better performance by collecting much more observations. Some meteorological data in each Korean city showed a relatively weak correlation with concentration, so it seems necessary to find variables that have causality or strong correlation within areas other than deep learning. For example, if spatial factors (spatial homogeneity, autocorrelation, etc.) in Chinese cities and Korean cities are added to the model as input variables, it is expected that the model will produce better performance by learning time and spatial features of data.

This research will continue to maximize the prediction performance of deep learning models by collecting observations and optimizing models, while applying new algorithms and adding other variables that have causality with concentration of PM_2.5 in terms of econometrics and spatial econometrics.